Brandyn / Techy
8ae43905dc
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
448 lines
34 KiB
Markdown
448 lines
34 KiB
Markdown
# Community Discord Media — plan
|
||
|
||
> Current state lives in **STATUS.md** (what runs, who does what, what's unfinished). This file is the design and the phase checklists.
|
||
|
||
One login for everything (Discord via Authentik), a watch page (Jellyfin), and the existing
|
||
ingest/browse/download stack (MediaMTX, copyparty, Syncthing) moved behind that login.
|
||
|
||
Host: OVH SYS-1 (Xeon E-2136, 32 GB, 2×4 TB mirror), Debian 13, Docker Compose, Caddy TLS.
|
||
Bundle: `vanguard-stream/` in this repo; live copy at `~/vanguard-stream` on the server.
|
||
|
||
## Where we are
|
||
|
||
| piece | state |
|
||
|-------|-------|
|
||
| MediaMTX ingest (RTMP 1935 / SRT 8890), per-streamer keys | live |
|
||
| copyparty browse/scrub/download/WebDAV at `https://ovh1.dumpstervigilantes.rip/` | live, local accounts |
|
||
| OBS dock (`/dock/`), dump-tonight (`/dump`) | live, dump uses its own password file |
|
||
| Syncthing clips intake (22000), admin UI at `/sync/` | live, Syncthing's own GUI login |
|
||
| normalize-sweep (one editing standard, raw files in hidden `.src/`) | live |
|
||
| nftables host firewall | live |
|
||
| Authentik (Discord login, groups) at `auth.` | live |
|
||
| Jellyfin at `watch.`, SSO via Authentik, direct play only | live |
|
||
| Vanguard Metadata plugin v0.1 | installed, libraries being switched to Shows |
|
||
| MariaDB (bot database) + `discord-bot` Python container (release installed in `/srv/bot/app`) | live; first release pending |
|
||
| Gitea (`git.`, MariaDB, ssh 2222) + Actions runner | live (local admin account; Authentik OIDC → Phase 10) |
|
||
|
||
## Decisions already made
|
||
|
||
- **Discord is the identity.** Authentik is the only self-hosted IdP that does Discord social login,
|
||
groups, OIDC *and* forward-auth for apps with no login of their own. Keycloak/Authelia/Pocket-ID don't fit.
|
||
- **Groups:** `streamers`, `editors`, `admins`. Access everywhere is by group, not by person.
|
||
- **Jellyfin is direct-play only** until a SYS-3 (Xeon E-2288G, Quick Sync). The E-2136 has no
|
||
hardware encoder; software transcoding tops out at ~2 concurrent viewers. Transcoding is off at
|
||
the server *and* per user.
|
||
- **MediaMTX stream keys stay machine credentials.** OBS can't do a Discord login. Keys are managed by
|
||
`configure.py`; the dock will fetch the logged-in streamer's key after Authentik login (phase 6).
|
||
- **Raw clipper files live in hidden `.src/` folders** so Jellyfin and copyparty show one item per clip.
|
||
- **Hostnames:** `auth.` and `watch.` are CNAMEs to the main host. Caddy routes by SNI, Let's Encrypt
|
||
follows CNAMEs. One A record to change if the IP ever moves.
|
||
- **Discord login only, no passwords anywhere** → no WebDAV (a mounted drive can't do a browser login).
|
||
Editors download from copyparty's web UI (zip/tar for folders). WebDAV stays *available* (Authentik
|
||
"Intercept header authentication" + per-user app passwords, no code change) if we ever choose it.
|
||
|
||
## Target layout
|
||
|
||
```
|
||
auth.dumpstervigilantes.rip Authentik — Discord login, groups, OIDC provider, forward-auth gate
|
||
watch.dumpstervigilantes.rip Jellyfin — libraries: streams / clips / finals, SSO plugin → Authentik
|
||
ovh1.dumpstervigilantes.rip Caddy
|
||
/ copyparty (forward-auth → Authentik; user+groups via headers; perms by @group)
|
||
/dock/ OBS dock (requires Authentik session; shows the caller's own stream key)
|
||
/dump dumpd (trusts X-Authentik-Username; admins may dump anyone)
|
||
/sync/ Syncthing (forward-auth, admins only)
|
||
rtmp://ovh1:1935 srt://ovh1:8890 MediaMTX (unchanged; per-streamer user/pass in stream key)
|
||
```
|
||
|
||
## Phases
|
||
|
||
### Phase 1 — DNS + Authentik/Jellyfin containers ✅ done → `addCommunityMediaServer.sh`
|
||
- [ ] CNAME `auth` → `ovh1.dumpstervigilantes.rip`
|
||
- [ ] CNAME `watch` → `ovh1.dumpstervigilantes.rip`
|
||
- [ ] from your machine: `scp -r bin docker-compose.yml configure.py install.sh addCommunityMediaServer.sh README.md CommunityDiscordMedia.md .env.example debian@ovh1…:~/vanguard-stream/`
|
||
- [ ] on the server: `cd ~/vanguard-stream && sudo install -m 0755 bin/normalize-sweep.sh /usr/local/bin/ && ./addCommunityMediaServer.sh`
|
||
— asks for the two hostnames (defaults derived from the main host), checks they resolve to this
|
||
box, writes Caddy blocks + `.env` (secrets) via `configure.py --hosts`, starts the containers,
|
||
waits for HTTPS on both names, then prints the Phase 2/3 steps with the real hostnames.
|
||
Re-runnable; remembers the hostnames.
|
||
- [ ] `docker compose ps` → 9 containers up; `free -h` sane
|
||
|
||
### Phase 2 — Authentik + Discord ✅ done (loose ends → Phase 8)
|
||
- [ ] `https://auth.HOST/if/flow/initial-setup/` → set `akadmin` password (store it somewhere safe)
|
||
- [ ] Discord developer portal → application → OAuth2 redirect `https://auth.HOST/source/oauth/callback/<slug>/`
|
||
(slug = the Authentik source slug, e.g. `discord`; they must match exactly)
|
||
- [ ] Authentik → Directory → Federation & Social login → Create → Discord OAuth Source
|
||
(slug `discord`; consumer key/secret = Discord client id/secret; user matching "link on unique
|
||
identifier" is fine; scopes empty; flows default-source-authentication / default-source-enrollment)
|
||
- [ ] rotate the Discord client secret after setup if it was ever pasted into a chat/screenshot
|
||
- [ ] Show the source on the login page: Flows & Stages → Stages → `default-authentication-identification`
|
||
→ Sources: select Discord, tick "Show sources' labels" (new sources are hidden until this is done)
|
||
- [ ] Make enrolled users Internal: Flows & Stages → Stages → `default-source-enrollment-write` →
|
||
User type: Internal (External users get "Interface can only be accessed by internal users")
|
||
- [ ] Directory → Groups: `streamers`, `editors`, `admins`
|
||
- [ ] Log in with Discord as a normal user; confirm the account is created and Internal; add yourself to `admins`
|
||
- [x] only members of our Discord server may log in — `authentik/discord-guild-policy.py`:
|
||
1. Discord Developer Mode → right-click server → Copy Server ID
|
||
2. Discord source → Scopes: `guilds`
|
||
3. Customization → Policies → Create Expression Policy `discord-guild-member`, paste the file,
|
||
set `ACCEPTED_GUILD_ID` / `GUILD_NAME`
|
||
4. bind it to `default-source-enrollment` **and** `default-source-authentication`
|
||
(flow → Policy / Group / User Bindings → Bind existing policy), then edit both flows →
|
||
Policy engine mode **all** (with `any`, the existing if-sso binding passes on its own)
|
||
5. test with a member and a non-member; existing users re-consent once at Discord
|
||
Later: a per-role variant uses `guilds.members.read` and `/users/@me/guilds/<id>/member`
|
||
to map Discord roles → Authentik groups automatically.
|
||
- [ ] Pin `AUTHENTIK_TAG` in `.env` to the version that was installed (don't float on upgrades)
|
||
|
||
### Phase 3 — Jellyfin ✅ done (loose ends → Phase 8)
|
||
- [ ] Wizard at `https://watch.HOST/`: admin user; libraries `/media/streams`, `/media/clips`,
|
||
`/media/finals` as "Home Videos", all metadata fetchers off
|
||
- [ ] Dashboard → Playback → Transcoding: hardware acceleration none; per user untick
|
||
"Allow video playback that requires transcoding"
|
||
- [ ] Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json` → Catalog → "SSO Authentication" → install → restart
|
||
- [ ] Authentik → Applications → Create with provider (OAuth2/OIDC) for Jellyfin, redirect
|
||
`https://watch.HOST/sso/OID/redirect/authentik`
|
||
- [ ] SSO-Auth → add provider `authentik`: OID endpoint `https://auth.HOST/application/o/<slug>/`,
|
||
client id/secret, roles claim `groups`, admin roles `admins`, enabled folders = 3 libraries,
|
||
default: transcoding disabled; Scheme Override `https`, Port Override EMPTY; Save the page
|
||
- [ ] direct test `https://watch.HOST/sso/OID/start/authentik` logs a Discord user in
|
||
(gotchas hit: unsaved provider → "Provider does not exist"; Port Override set → redirect gets `:443`
|
||
→ Authentik mismatch; opening the /redirect/ URL by hand → "missing state")
|
||
- [ ] login page: Discord button first, local form behind "Admin login" — Dashboard → General → Branding:
|
||
disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname), CSS = `jellyfin/branding.css`
|
||
- [ ] button works for a Discord user in `editors`
|
||
|
||
### Phase 4 — "Vanguard Metadata" Jellyfin plugin (NEXT — usability for everyone)
|
||
|
||
Decision: this lives **inside Jellyfin as a metadata provider plugin**, not as server scripts and
|
||
timers. Files stay exactly where they land (Syncthing mirrors the gamer's folder; streams stay in
|
||
`streams/<user>/<Title>/`). No sidecar files, no symlink view, no moves. Jellyfin asks the plugin
|
||
for metadata and the plugin answers from the filename, the folder, Discord and Authentik.
|
||
|
||
**What the plugin provides** (C#, Jellyfin plugin SDK, targets the installed server's ABI)
|
||
- `IRemoteMetadataProvider<Series>` — show = the `<user>` folder: display name from Discord,
|
||
overview from the member's roles/bio
|
||
- `IRemoteImageProvider` (Series) — poster = the Discord avatar
|
||
- `IRemoteMetadataProvider<Season>` — season **name** for a season number (policy / override / event)
|
||
- `IRemoteMetadataProvider<Episode>` — **season + episode numbers from the timestamp** (rolling
|
||
policy below), title from the Discord event overlapping the recording (fallback: the dock title =
|
||
folder name), overview = event description + duration + streamer, premiere date
|
||
- `IScheduledTask` "Discord snapshot" — members + scheduled events into the plugin's data store
|
||
every 5 min (Discord only exposes upcoming/active events; history must be kept)
|
||
- `IScheduledTask` "Season roll" — nightly after 04:00 Eastern: refresh metadata on items whose
|
||
bucket changed (weekday → week → month → year). Same path, same item ID → watched state kept.
|
||
- configuration page: Discord bot token + guild ID, Authentik URL + API token (streamer → Discord ID
|
||
via the OAuth source connection), broadcast-day hour, rolling thresholds, season name overrides,
|
||
provider on/off per library
|
||
|
||
**Timestamp precedence** (plugin-side): bot `clip.json` sidecar if present (Medal/Twitch/YouTube
|
||
creation time) → container `creation_time` (Jellyfin's MediaInfo exposes it) → MediaMTX filename →
|
||
file date. Normalised to America/New_York.
|
||
|
||
**Rolling seasons by tag** (ParentIndexNumber/IndexNumber set by the provider; pure function of
|
||
timestamp + policy):
|
||
|
||
| age of item | season # | season name |
|
||
|---|---|---|
|
||
| current week | `YYYYMMDD` | "Thursday 30 Oct" |
|
||
| current month, past week | `YYYYWW` (ISO) | "Week 44" |
|
||
| current year, past month | `YYYYMM` | "October 2026" |
|
||
| past year | `YYYY` | "2026" |
|
||
|
||
episode # derived from the timestamp (`DDHHMM`-style; stable; same-minute collisions get a digit).
|
||
Broadcast day rolls at 04:00. Season names: generated → override table → Discord event series name
|
||
when every item in the bucket belongs to one.
|
||
|
||
**Libraries**: Shows type for `streams` and `clips` with this plugin as the only Series/Season/Episode
|
||
provider (TheTVDB/TMDB off), Movies type for `finals` (plugin optional). Jellyfin's scanner treats
|
||
`<user>` as the series and any file below it as an episode; the plugin supplies the numbers.
|
||
|
||
**Build & ship**: .NET 8 SDK, `Jellyfin.Controller` matching the server's major version; CI builds
|
||
the DLL + zip; served from our Caddy as a plugin repository (`/plugins/manifest.json`) so installs
|
||
and updates go through Jellyfin's own Catalog. Cost: C#, and a rebuild on each Jellyfin major.
|
||
|
||
Status: **v0.1.1.0** — v0.1.0 installed and **verified**: Jellyfin honours the provider's season numbers
|
||
(day seasons appeared on a Shows library). v0.1.1 adds the playback-policy task. Providers, Discord/Authentik
|
||
clients, event history, season-roll + snapshot tasks, Discord feed with frame-grab attachment, config page;
|
||
10 unit tests.
|
||
|
||
Checklist:
|
||
- [x] skeleton + core season policy (tests) + providers + tasks + feed → `./package.sh` → `dist/VanguardMetadata_0.1.0.0.zip`
|
||
- [ ] Discord dev portal → the existing Authentik application → **Bot** → Reset Token; enable **Server Members**
|
||
intent; OAuth2 URL generator: scope `bot`, perms View Channels + Send Messages + Embed Links + Attach Files;
|
||
invite to the server
|
||
- [ ] Authentik: read-only service account + API token for the plugin
|
||
- [ ] install the zip (README in the plugin folder), restart Jellyfin, configure the plugin page
|
||
- [x] libraries → Shows type; providers: only Vanguard; scan — day seasons appeared
|
||
- [ ] install v0.1.1 (`./installJellyfinPlugin.sh`), run "enforce playback policy" once, check a user's policy
|
||
- [ ] verify: seasons appear as "Thursday, Oct 29" / "Week 44" / "October 2026"; poster = avatar;
|
||
a new file posts to the feed channel with the frame grab
|
||
- [ ] Medal local filename pattern → add to `ItemTimestamp` (need a sample filename)
|
||
- [ ] later: serve `dist/` at `/plugins/` via Caddy as a plugin repo; theme CSS
|
||
|
||
### Phase 5 — copyparty behind Authentik ✅ live
|
||
- [ ] Authentik → Providers → Proxy provider, *forward auth (single application)*, external host
|
||
`https://ovh1.HOST`; Application bound to it; add to the embedded outpost
|
||
- [ ] Caddy: `forward_auth 127.0.0.1:9000 { uri /outpost.goauthentik.io/auth/caddy; copy_headers X-Authentik-Username X-Authentik-Groups … }`
|
||
on `/`, `/dock*`, `/dump*`, `/sync*`; `reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000`
|
||
- [ ] copyparty: `idp-h-usr: X-Authentik-Username`, `idp-h-grp: X-Authentik-Groups`, `xff-src` = Caddy only;
|
||
volume accs become `r: @editors`, `rw: @editors` (finals), `rwmd: @media-admins` on every volume incl. trash
|
||
(new group `media-admins`; techy joins it via a Discord role in the roles→groups table or a manual add —
|
||
replaces today's workaround of logging in as the local `admin` account to modify streams/clips)
|
||
- [ ] keep the local accounts in `copyparty.conf` until an `editors` Discord user has browsed and
|
||
downloaded through the new path; then remove them (`configure.py` drops the editors section)
|
||
- [x] decided: no WebDAV (Discord login only); browser downloads
|
||
- [ ] policy on the Application: only `editors` + `admins` + `streamers` may pass
|
||
|
||
### Phase 6 — Syncthing UI + dump behind Authentik ✅ live
|
||
- [ ] `/sync/`: forward-auth, policy `admins` only; Syncthing GUI auth can stay as a second layer
|
||
- [ ] `dumpd.py`: read `X-Authentik-Username` / `X-Authentik-Groups` from Caddy instead of basic auth;
|
||
admin = member of `admins`; drop `dump-users.txt`
|
||
- [ ] dock: `Dump tonight` sends no credentials; session cookie carries identity
|
||
|
||
### Phase 7 — dock polish + cleanup ✅ live (dock mints tokens; no stored credentials)
|
||
- [ ] `dumpd.py` gains `GET /me` → `{user, groups, stream_key}` for the logged-in streamer;
|
||
dock fills OBS settings from it — nobody types a stream password again
|
||
- [ ] `configure.py` shrinks: hostnames + streamers only (editors and admin live in Authentik)
|
||
- [ ] README: replace the per-app login notes with "log in with Discord"
|
||
|
||
### Phase 8 — hardening (loose ends from phases 2–3, plus backups)
|
||
- [ ] Jellyfin login branding: Discord button first, "Admin login" reveals the local form
|
||
(`jellyfin/branding-disclaimer.html` + `jellyfin/branding.css`)
|
||
- [ ] group check: a Discord user in `editors` but not `admins` sees the three libraries and is NOT a
|
||
Jellyfin administrator; one in `admins` is
|
||
- [x] per-user transcoding OFF for every user, automatically: the Vanguard plugin's "enforce playback
|
||
policy" task (startup + every 30 min) sets direct-play-only on all non-admin users, SSO-created
|
||
ones included. Config toggle `EnforceDirectPlayOnly`; turn off on the SYS-3
|
||
- [ ] pin Authentik: `.env` → `AUTHENTIK_TAG=<tag now running>` (`docker inspect authentik-server --format '{{.Config.Image}}'`)
|
||
- [x] guild restriction so only members of our Discord server can log in
|
||
- [ ] rotate the Discord client secret that appeared in a screenshot; paste the new one into the source
|
||
- [ ] nightly `pg_dump` of the Authentik database into `/srv/appdata/authentik/backup/` (timer), keep 14 days;
|
||
Jellyfin config and Authentik media are plain directories, covered by the archive/backup of `/srv/appdata`
|
||
- [ ] Jellyfin: library "real time monitoring" on, metadata fetchers confirmed off, scheduled scan daily
|
||
|
||
### Discord bot (separate dev) — runtime provided by the stack
|
||
- `mariadb` container, db `vanguardbot` / user `bot`, loopback only (ssh tunnel for dev access)
|
||
- `discord-bot` container: python 3.12 + ffmpeg + curl, and no bot dependencies — its CI builds a release
|
||
bundle (code + `vendor/` wheels) that dumpd installs into `/srv/bot/app`, bind-mounted at `/app`;
|
||
env gives token, guild, DB creds, rw `/media/clips`, `/data` scratch; restarts on exit
|
||
- web UI at `bot.` behind Caddy + Authentik forward-auth (first use of the forward-auth pattern that
|
||
Phase 5 applies to copyparty); the bot trusts `X-Authentik-*` headers and implements no login
|
||
- `bot/AGENT.md` is the build brief for the agent writing the bot; `bot/README.md` the ops cheat-sheet. The clip-submissions feature (Option below) is a natural home
|
||
for this bot: write `clips/<user>/<name>.mp4` + `<name>.json` sidecar and the Jellyfin plugin picks it up.
|
||
- [ ] CNAME `bot` → main host; `configure.py --full` (asks the hostname, bot token + guild) → `docker compose up -d --build`
|
||
- [ ] Authentik: domain-level Proxy Provider (covers live/bot/main); Caddy requires group `bot-users` on the bot host (`BOT_REQUIRE_GROUP`)
|
||
- [ ] Discord roles → Authentik groups: `authentik/discord-roles-to-groups.py` on both source flows (scope `guilds.members.read`)
|
||
- [x] the bot repo's CI builds a release bundle and a push to `main` installs it (signed, via dumpd `/deploy-bot`)
|
||
|
||
### Phase 9 — Gitea + Actions runner
|
||
- `gitea` on MariaDB (db/user from `mariadb/init/10-gitea.sh` on first init, else `./createGiteaDb.sh`),
|
||
`git.` via Caddy, ssh 2222 (firewall opened), registration closed, Actions enabled
|
||
- `act_runner` with the Docker socket; labels ubuntu-latest / ubuntu-22.04 / dotnet / python
|
||
- [ ] CNAME `git` → main host; `./configure.py --full` (asks the hostname) or `--hosts AUTH WATCH GIT`
|
||
- [ ] https://git.HOST/ install page → admin account
|
||
- [ ] `./registerGiteaRunner.sh`; confirm the runner is idle under Site Administration → Actions → Runners
|
||
- [ ] create users (or OIDC via Authentik, same pattern as Jellyfin: Gitea → Authentication Sources → OpenID Connect)
|
||
- [ ] first use: push `vanguard-stream` itself; a workflow that runs `package.sh` on the `dotnet` label and
|
||
publishes the plugin zip is the obvious first pipeline
|
||
- risk: the runner's Docker socket is root on the host for anyone who can push a workflow
|
||
|
||
### Phase 10 — Gitea login via Authentik (OIDC) ✅ live, verified
|
||
Same pattern as Jellyfin: Authentik is the identity, Gitea trusts it.
|
||
- [ ] Authentik → Applications → Create with provider → OAuth2/OpenID: name `gitea`, redirect
|
||
`https://git.HOST/user/oauth2/authentik/callback`, authorization flow implicit-consent, scopes
|
||
openid email profile (+ `groups` claim via the default profile mapping)
|
||
- [ ] Gitea → Site Administration → Identity & Access → Authentication Sources → Add → OAuth2,
|
||
provider OpenID Connect, name `authentik` (must match the callback path above),
|
||
client ID/secret, discovery URL `https://auth.HOST/application/o/<slug>/.well-known/openid-configuration`,
|
||
"Skip local 2FA", claim mapping: group claim `groups` → Gitea admin group `admins`
|
||
- [x] Gitea env: `DISABLE_REGISTRATION=false` + `ALLOW_ONLY_EXTERNAL_REGISTRATION=true` (local self-signup stays
|
||
off), `ENABLE_AUTO_REGISTRATION`, `ACCOUNT_LINKING=auto`, `USERNAME=nickname`, `UPDATE_AVATAR`
|
||
- [ ] optional: hide the password form on the login page (custom template) so Discord is the only visible way in
|
||
- [ ] Authentik groups `git-users` and `git-admins` bound on the `Gitea` application; Gitea admin claim = `git-admins`;
|
||
nesting `authentik Admins` → `admins` → `git-admins` so admins inherit it
|
||
- [ ] test with a non-admin Discord user; confirm admin mapping via `admins` group
|
||
|
||
### Phase 11 — MediaMTX auth through Authentik (no stream keys anywhere) ✅ live
|
||
Implemented variant: **short-lived HMAC tokens minted by dumpd** for Discord-authenticated members of
|
||
`streamers` (no Authentik app passwords — "Discord only" decision). Dock and portal mint a token per Go Live;
|
||
manual setups get it from the dock's accordion. MediaMTX `authMethod: http` → dumpd `/mtx-auth` on 8081
|
||
(Docker bridge only). Hot-reloaded: no MediaMTX restart.
|
||
Goal: nobody holds a MediaMTX password. "May publish" = Authentik identity + `streamers` group (filled by
|
||
Discord roles); the publish path must start with the user's own name.
|
||
- MediaMTX `authMethod: http`, `authHTTPAddress: http://127.0.0.1:8080/mtx-auth` (dumpd becomes the
|
||
arbiter; reads stay open to `any`, Caddy gates viewers). MediaMTX posts user/pass/token/action/path/ip.
|
||
- **Browser (Go Live, WHEP)**: Caddy forwards Authentik's `X-Authentik-Jwt` as `Authorization: Bearer` on
|
||
the `/webrtc` hop. dumpd verifies the JWT against Authentik's JWKS (`/application/o/<slug>/jwks/`),
|
||
reads `preferred_username` + `groups`, allows publish iff `path` starts with `<username>/` and user ∈ streamers.
|
||
No `/me/stream-key`, no `streamers.json`, nothing in the browser.
|
||
- **OBS**: streamer creates an **app password** in their Authentik profile and pastes it in the dock once;
|
||
dumpd validates Basic `user:app-password` against Authentik (outpost basic-auth endpoint or the API),
|
||
same path rule. Revoking the app password revokes the stream key.
|
||
- configure.py loses the streamers section (folders are created on first publish by MediaMTX anyway;
|
||
the sweep and Jellyfin cope). Dock: "paste your Authentik app password" instead of a stream password.
|
||
- [ ] dumpd `/mtx-auth` (JWT via JWKS, Basic via Authentik, path rule, 5 s cache)
|
||
- [ ] Caddy: `header_up Authorization "Bearer {http.request.header.X-Authentik-Jwt}"` on `/webrtc/*` (replaces X-Stream-Auth)
|
||
- [ ] MediaMTX `authMethod: http`; keep `any` read; drop `authInternalUsers`
|
||
- [ ] dock + README; remove streamers from configure.py; delete `streamers.json`
|
||
- [ ] test: OBS with an app password; Go Live with no key; a non-streamer gets denied
|
||
|
||
### Phase 12 — frame-accurate timecode for multicam via an overscan band ✅ live, verified (OBS: strip, −3 ms drift, 0 gaps)
|
||
Target: clips from several streamers align to the frame (16.7 ms @60) on a multicam timeline; the game
|
||
image is untouched and finals carry nothing. Reference clock = the server; streamer PC clocks are never trusted.
|
||
|
||
Facts that shape it: within a recording, frame timing is the encoder's capture clock (network lag only shifts
|
||
*arrival*, which is measured once); but a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h), so a
|
||
permanent reference is needed for long sessions, not just a start burst.
|
||
|
||
- **Overscan band**: the published canvas is the game image plus 16 extra rows at the bottom. The band holds a
|
||
machine-readable strip (black/white cells: server time in ms + checksum + framing, cell size scaled to the
|
||
canvas) and a small human-readable server clock, **continuously**.
|
||
- Go Live: canvas composite = video (W×H) + band → W×(H+16); the `/clock/` logic runs in-page.
|
||
- OBS: the dock (obs-websocket) offers "add overscan band": SetVideoSettings canvas/output to H+16 while not
|
||
streaming, CreateInput `browser_source` → `/clock/` positioned in the band, toggle/remove. The browser
|
||
source shares the dock's cookies (Authentik). Opt-out = coarse timecode only.
|
||
- `/clock/` measures its offset to the server clock over HTTPS (±few ms) and shows **server** time.
|
||
- **Sweep**: when publishing a stream: extract frames at the start, middle and end; locate + decode the strip;
|
||
frame-0 capture time = median(decoded − pts) at the start (frame-accurate) → QuickTime `tmcd` track
|
||
(time of day, Eastern, file fps) + `creation_time` (UTC) in the existing lossless remux; a `.tc.json`
|
||
sidecar with the samples records drift over the file. No strip → server arrival time (coarse), logged.
|
||
Strip present → the band is cropped off in a re-encode (decided 2026-11-01: finals = picture + audio + tmcd,
|
||
nothing else); no strip → lossless remux, never crop.
|
||
- **Viewing**: live portal crops the band on the live picture (canvas/overflow fix pending); recordings are
|
||
already at game resolution.
|
||
- MediaMTX `recordPath` with `%f` (microseconds) for the fallback; plugin `ItemTimestamp` prefers `creation_time`.
|
||
- [x] `/clock/` page: offset measurement, strip encoder, readable clock, band layout
|
||
- [x] dock: overscan band setup + browser source via obs-websocket; opt-out
|
||
- [x] portal Go Live: band composite (toggle); portal viewer: CSS crop of the band
|
||
- [x] sweep: `bin/vgstrip.py` (ffmpeg frames → decode → offset + drift), tmcd + creation_time, `<stem>.json` sidecar (read by the plugin)
|
||
- [x] MediaMTX `%f`; plugin timestamp update; README editor workflow (multicam: timecode → audio cross-check;
|
||
export at 2560×1440)
|
||
- rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
|
||
|
||
### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (NEXT)
|
||
|
||
**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
|
||
Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
|
||
host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
|
||
VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
|
||
expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
|
||
|
||
- [ ] CNAME `diag` → the main host
|
||
- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
|
||
- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
|
||
`-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
|
||
strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
|
||
- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
|
||
trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
|
||
`vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
|
||
filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
|
||
(JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
|
||
multiline + regex for jellyfin).
|
||
- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
|
||
and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
|
||
alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
|
||
json files directly avoids both and loses the container labels, which is most of the value.
|
||
- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
|
||
longer than the rest would mean separate tenants — not worth it unless an audit needs it.
|
||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||
would consume them. Not required for logs.
|
||
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
|
||
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
|
||
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
|
||
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
|
||
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
|
||
scraping the four taps that are already live. Full picture: four containers.
|
||
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
|
||
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
|
||
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
|
||
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
|
||
version for thresholds, grouping and silencing, not a prerequisite for being warned.
|
||
|
||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||
Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
|
||
purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
|
||
logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
|
||
|
||
### Phase 14 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
|
||
- server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved
|
||
- `vanguard-deploy pull|apply|rollback|pending|status` applies only the changed parts; `check.sh` gates it;
|
||
MediaMTX restarts are deferred while anything is live (`deploy-pending.timer`)
|
||
- Gitea push webhook → dumpd `/deploy` (bridge port, HMAC) → `vanguard-deploy pull` when `DEPLOY_AUTO=1`
|
||
- CI (draft): `check.sh` + plugin tests/build on the runner; the runner never touches the host
|
||
- [ ] run `switchToClone.sh key` / add deploy key / `switchToClone.sh clone` on the server
|
||
- [ ] `./configure.py --apply` once (installs vanguard-deploy, deploy.env, the pending timer)
|
||
- [ ] add the Gitea webhook (secret from `.env`); watch `journalctl -u dumpd | grep deploy` for a few pushes
|
||
- [ ] when trusted: `DEPLOY_AUTO=1` in `.env` + `./configure.py --apply`; move `ci.yml` into `.gitea/workflows/`
|
||
|
||
### Option — more clip sources (Medal, Twitch/YouTube/Kick clips, Discord uploads)
|
||
|
||
Today the only clip intake is a local clipper folder synced by Syncthing. Members without that
|
||
setup have no way in. Two additions cover everyone, and neither depends on Authentik.
|
||
|
||
**A. `#clip-submissions` Discord bot** (one container, outbound-only, no new ports)
|
||
- member posts a Medal.tv / Twitch clip / YouTube / Kick / Streamable link, or drops a file
|
||
- bot maps the poster's Discord username → `/srv/media/clips/<username>/` (same identity Authentik uses)
|
||
- downloads at best quality with **yt-dlp** (has extractors for all of the above incl. Medal)
|
||
- writes `clip.json` beside it: source URL, title, submitter, timestamp — context for editors
|
||
- reacts ✅ / ❌ on the message; replies with the reason on failure
|
||
- normalize-sweep handles whatever arrives (Medal's HEVC/VFR etc.)
|
||
- config: bot token + channel id in `.env`; yt-dlp pinned, self-updates weekly (extractors rot)
|
||
- quality note: Medal/Twitch/YouTube deliver their re-encode (~1080p). Fine for clips; originals
|
||
still come via Syncthing or upload below.
|
||
- skipped on purpose: polling Medal profiles via its API — needs per-member IDs, pulls clips nobody
|
||
submitted, and only returns embed URLs; the bot covers it with explicit intent.
|
||
|
||
**B. Web upload for members (after Phase 5)**
|
||
- copyparty per-user volume `clips/${u}` writable by its owner, readable by `editors`/`admins`
|
||
- anyone who can log in with Discord drags an original into the browser or WebDAV — full quality,
|
||
no bot, no install
|
||
|
||
| source | path in | quality |
|
||
|--------|---------|---------|
|
||
| local clipper + Syncthing | automatic | original |
|
||
| Medal / Twitch / YouTube / Kick link | bot (A) | platform re-encode |
|
||
| Discord attachment | bot (A) | original, Discord size cap |
|
||
| file in hand | web upload (B) | original |
|
||
|
||
Checklist if adopted:
|
||
- [ ] Discord developer portal → bot user, Message Content intent, invite to the server with
|
||
read/react/reply in `#clip-submissions`
|
||
- [ ] `bot/` in the bundle: python + discord.py + yt-dlp, runs as uid 1000, mounts `/srv/media/clips`
|
||
- [ ] `.env`: `DISCORD_BOT_TOKEN`, `CLIPS_CHANNEL_ID`
|
||
- [ ] per-user clips volume in `copyparty.conf` once forward-auth is live (Phase 5)
|
||
|
||
### Later — when it hurts
|
||
|
||
- [ ] **Off-site gap filling** (instead of burning the SYS-1's CPU): streams whose sidecar reports
|
||
`gaps.total_missing_ms` ≥ threshold are re-encoded elsewhere with a hardware encoder.
|
||
- the sweep keeps the raw recording in `.src/` and the sidecar flags it (`needs_fill: true`)
|
||
- a nightly job on the home box (RTX 2080 Ti, NVENC) or a Spot GPU instance pulls flagged raws over
|
||
SFTP, re-encodes to CFR with duplicated frames (same editing standard, `tmcd` + `creation_time`
|
||
carried from the sidecar), pushes the result back over the published file, clears the flag
|
||
- Jellyfin keeps the same path → item identity and watched state survive; the plugin re-reads metadata
|
||
- until then: `FILL_GAPS_MS` stays off; players hold frames across gaps, editors see the gap list
|
||
- [ ] **SYS-3 (E-2288G, Quick Sync)** — the 2×6 TB disk option is currently unavailable; re-check and swap when
|
||
it returns. Then: Jellyfin transcoding on, sweep re-encodes in minutes, `FILL_GAPS_MS` can simply be on.
|
||
- [ ] **SYS-3 (E-2288G)**: Quick Sync → Jellyfin transcoding on, sweep re-encodes in minutes.
|
||
Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
|
||
- [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
|
||
|
||
|
||
## Risks / gotchas
|
||
|
||
- **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the
|
||
Authentik path is proven. Caddy config errors stop *all* sites — `docker logs caddy` first.
|
||
- **Authentik outpost header names** differ by version (`X-Authentik-Username`, `X-Authentik-Groups`,
|
||
`X-Authentik-Email`). Verify with a test route that echoes headers before wiring copyparty.
|
||
- **Jellyfin SSO plugin** creates users with transcoding allowed by default unless the provider's
|
||
defaults are set — set them, or the per-user toggle has to be fixed after every first login.
|
||
- **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
|
||
- **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
|
||
- **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
|
||
|
||
## Monthly cost
|
||
Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.
|