Vanguard media stack: ingest, copyparty, Syncthing, normalize sweep, Authentik, Jellyfin + Vanguard Metadata plugin, MariaDB, Gitea, bot runtime

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -0,0 +1,317 @@
+# Community Discord Media — plan
+
+> Current state lives in **STATUS.md** (what runs, who does what, what's unfinished). This file is the design and the phase checklists.
+
+One login for everything (Discord via Authentik), a watch page (Jellyfin), and the existing
+ingest/browse/download stack (MediaMTX, copyparty, Syncthing) moved behind that login.
+
+Host: OVH SYS-1 (Xeon E-2136, 32 GB, 2×4 TB mirror), Debian 13, Docker Compose, Caddy TLS.
+Bundle: `vanguard-stream/` in this repo; live copy at `~/vanguard-stream` on the server.
+
+## Where we are
+
+| piece | state |
+|-------|-------|
+| MediaMTX ingest (RTMP 1935 / SRT 8890), per-streamer keys | live |
+| copyparty browse/scrub/download/WebDAV at `https://ovh1.dumpstervigilantes.rip/` | live, local accounts |
+| OBS dock (`/dock/`), dump-tonight (`/dump`) | live, dump uses its own password file |
+| Syncthing clips intake (22000), admin UI at `/sync/` | live, Syncthing's own GUI login |
+| normalize-sweep (one editing standard, raw files in hidden `.src/`) | live |
+| nftables host firewall | live |
+| Authentik (Discord login, groups) at `auth.` | live |
+| Jellyfin at `watch.`, SSO via Authentik, direct play only | live |
+| Vanguard Metadata plugin v0.1 | installed, libraries being switched to Shows |
+| MariaDB (bot database) + `discord-bot` Python container (`bot/app`) | **in bundle, not yet applied** |
+| Gitea (`git.`, MariaDB, ssh 2222) + Actions runner | live (local admin account; Authentik OIDC → Phase 10) |
+
+## Decisions already made
+
+- **Discord is the identity.** Authentik is the only self-hosted IdP that does Discord social login,
+  groups, OIDC *and* forward-auth for apps with no login of their own. Keycloak/Authelia/Pocket-ID don't fit.
+- **Groups:** `streamers`, `editors`, `admins`. Access everywhere is by group, not by person.
+- **Jellyfin is direct-play only** until a SYS-3 (Xeon E-2288G, Quick Sync). The E-2136 has no
+  hardware encoder; software transcoding tops out at ~2 concurrent viewers. Transcoding is off at
+  the server *and* per user.
+- **MediaMTX stream keys stay machine credentials.** OBS can't do a Discord login. Keys are managed by
+  `configure.py`; the dock will fetch the logged-in streamer's key after Authentik login (phase 6).
+- **Raw clipper files live in hidden `.src/` folders** so Jellyfin and copyparty show one item per clip.
+- **Hostnames:** `auth.` and `watch.` are CNAMEs to the main host. Caddy routes by SNI, Let's Encrypt
+  follows CNAMEs. One A record to change if the IP ever moves.
+- **WebDAV through Authentik** uses Authentik app passwords as HTTP basic auth (a mounted drive
+  can't do a browser login).
+
+## Target layout
+
+```
+auth.dumpstervigilantes.rip   Authentik  — Discord login, groups, OIDC provider, forward-auth gate
+watch.dumpstervigilantes.rip  Jellyfin   — libraries: streams / clips / finals, SSO plugin → Authentik
+ovh1.dumpstervigilantes.rip   Caddy
+   /            copyparty   (forward-auth → Authentik; user+groups via headers; perms by @group)
+   /dock/       OBS dock    (requires Authentik session; shows the caller's own stream key)
+   /dump        dumpd       (trusts X-Authentik-Username; admins may dump anyone)
+   /sync/       Syncthing   (forward-auth, admins only)
+rtmp://ovh1:1935  srt://ovh1:8890   MediaMTX (unchanged; per-streamer user/pass in stream key)
+```
+
+## Phases
+
+### Phase 1 — DNS + Authentik/Jellyfin containers  ✅ done  → `addCommunityMediaServer.sh`
+- [ ] CNAME `auth` → `ovh1.dumpstervigilantes.rip`
+- [ ] CNAME `watch` → `ovh1.dumpstervigilantes.rip`
+- [ ] from your machine: `scp -r bin docker-compose.yml configure.py install.sh addCommunityMediaServer.sh README.md CommunityDiscordMedia.md .env.example debian@ovh1…:~/vanguard-stream/`
+- [ ] on the server: `cd ~/vanguard-stream && sudo install -m 0755 bin/normalize-sweep.sh /usr/local/bin/ && ./addCommunityMediaServer.sh`
+      — asks for the two hostnames (defaults derived from the main host), checks they resolve to this
+      box, writes Caddy blocks + `.env` (secrets) via `configure.py --hosts`, starts the containers,
+      waits for HTTPS on both names, then prints the Phase 2/3 steps with the real hostnames.
+      Re-runnable; remembers the hostnames.
+- [ ] `docker compose ps` → 9 containers up; `free -h` sane
+
+### Phase 2 — Authentik + Discord  ✅ done (loose ends → Phase 8)
+- [ ] `https://auth.HOST/if/flow/initial-setup/` → set `akadmin` password (store it somewhere safe)
+- [ ] Discord developer portal → application → OAuth2 redirect `https://auth.HOST/source/oauth/callback/<slug>/`
+      (slug = the Authentik source slug, e.g. `discord`; they must match exactly)
+- [ ] Authentik → Directory → Federation & Social login → Create → Discord OAuth Source
+      (slug `discord`; consumer key/secret = Discord client id/secret; user matching "link on unique
+      identifier" is fine; scopes empty; flows default-source-authentication / default-source-enrollment)
+- [ ] rotate the Discord client secret after setup if it was ever pasted into a chat/screenshot
+- [ ] Show the source on the login page: Flows & Stages → Stages → `default-authentication-identification`
+      → Sources: select Discord, tick "Show sources' labels" (new sources are hidden until this is done)
+- [ ] Make enrolled users Internal: Flows & Stages → Stages → `default-source-enrollment-write` →
+      User type: Internal (External users get "Interface can only be accessed by internal users")
+- [ ] Directory → Groups: `streamers`, `editors`, `admins`
+- [ ] Log in with Discord as a normal user; confirm the account is created and Internal; add yourself to `admins`
+- [x] only members of our Discord server may log in — `authentik/discord-guild-policy.py`:
+      1. Discord Developer Mode → right-click server → Copy Server ID
+      2. Discord source → Scopes: `guilds`
+      3. Customization → Policies → Create Expression Policy `discord-guild-member`, paste the file,
+         set `ACCEPTED_GUILD_ID` / `GUILD_NAME`
+      4. bind it to `default-source-enrollment` **and** `default-source-authentication`
+         (flow → Policy / Group / User Bindings → Bind existing policy), then edit both flows →
+         Policy engine mode **all** (with `any`, the existing if-sso binding passes on its own)
+      5. test with a member and a non-member; existing users re-consent once at Discord
+      Later: a per-role variant uses `guilds.members.read` and `/users/@me/guilds/<id>/member`
+      to map Discord roles → Authentik groups automatically.
+- [ ] Pin `AUTHENTIK_TAG` in `.env` to the version that was installed (don't float on upgrades)
+
+### Phase 3 — Jellyfin  ✅ done (loose ends → Phase 8)
+- [ ] Wizard at `https://watch.HOST/`: admin user; libraries `/media/streams`, `/media/clips`,
+      `/media/finals` as "Home Videos", all metadata fetchers off
+- [ ] Dashboard → Playback → Transcoding: hardware acceleration none; per user untick
+      "Allow video playback that requires transcoding"
+- [ ] Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json` → Catalog → "SSO Authentication" → install → restart
+- [ ] Authentik → Applications → Create with provider (OAuth2/OIDC) for Jellyfin, redirect
+      `https://watch.HOST/sso/OID/redirect/authentik`
+- [ ] SSO-Auth → add provider `authentik`: OID endpoint `https://auth.HOST/application/o/<slug>/`,
+      client id/secret, roles claim `groups`, admin roles `admins`, enabled folders = 3 libraries,
+      default: transcoding disabled; Scheme Override `https`, Port Override EMPTY; Save the page
+- [ ] direct test `https://watch.HOST/sso/OID/start/authentik` logs a Discord user in
+      (gotchas hit: unsaved provider → "Provider does not exist"; Port Override set → redirect gets `:443`
+      → Authentik mismatch; opening the /redirect/ URL by hand → "missing state")
+- [ ] login page: Discord button first, local form behind "Admin login" — Dashboard → General → Branding:
+      disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname), CSS = `jellyfin/branding.css`
+- [ ] button works for a Discord user in `editors`
+
+### Phase 4 — "Vanguard Metadata" Jellyfin plugin  (NEXT — usability for everyone)
+
+Decision: this lives **inside Jellyfin as a metadata provider plugin**, not as server scripts and
+timers. Files stay exactly where they land (Syncthing mirrors the gamer's folder; streams stay in
+`streams/<user>/<Title>/`). No sidecar files, no symlink view, no moves. Jellyfin asks the plugin
+for metadata and the plugin answers from the filename, the folder, Discord and Authentik.
+
+**What the plugin provides** (C#, Jellyfin plugin SDK, targets the installed server's ABI)
+- `IRemoteMetadataProvider<Series>` — show = the `<user>` folder: display name from Discord,
+  overview from the member's roles/bio
+- `IRemoteImageProvider` (Series) — poster = the Discord avatar
+- `IRemoteMetadataProvider<Season>` — season **name** for a season number (policy / override / event)
+- `IRemoteMetadataProvider<Episode>` — **season + episode numbers from the timestamp** (rolling
+  policy below), title from the Discord event overlapping the recording (fallback: the dock title =
+  folder name), overview = event description + duration + streamer, premiere date
+- `IScheduledTask` "Discord snapshot" — members + scheduled events into the plugin's data store
+  every 5 min (Discord only exposes upcoming/active events; history must be kept)
+- `IScheduledTask` "Season roll" — nightly after 04:00 Eastern: refresh metadata on items whose
+  bucket changed (weekday → week → month → year). Same path, same item ID → watched state kept.
+- configuration page: Discord bot token + guild ID, Authentik URL + API token (streamer → Discord ID
+  via the OAuth source connection), broadcast-day hour, rolling thresholds, season name overrides,
+  provider on/off per library
+
+**Timestamp precedence** (plugin-side): bot `clip.json` sidecar if present (Medal/Twitch/YouTube
+creation time) → container `creation_time` (Jellyfin's MediaInfo exposes it) → MediaMTX filename →
+file date. Normalised to America/New_York.
+
+**Rolling seasons by tag** (ParentIndexNumber/IndexNumber set by the provider; pure function of
+timestamp + policy):
+
+| age of item | season # | season name |
+|---|---|---|
+| current week | `YYYYMMDD` | "Thursday 30 Oct" |
+| current month, past week | `YYYYWW` (ISO) | "Week 44" |
+| current year, past month | `YYYYMM` | "October 2026" |
+| past year | `YYYY` | "2026" |
+
+episode # derived from the timestamp (`DDHHMM`-style; stable; same-minute collisions get a digit).
+Broadcast day rolls at 04:00. Season names: generated → override table → Discord event series name
+when every item in the bucket belongs to one.
+
+**Libraries**: Shows type for `streams` and `clips` with this plugin as the only Series/Season/Episode
+provider (TheTVDB/TMDB off), Movies type for `finals` (plugin optional). Jellyfin's scanner treats
+`<user>` as the series and any file below it as an episode; the plugin supplies the numbers.
+
+**Build & ship**: .NET 8 SDK, `Jellyfin.Controller` matching the server's major version; CI builds
+the DLL + zip; served from our Caddy as a plugin repository (`/plugins/manifest.json`) so installs
+and updates go through Jellyfin's own Catalog. Cost: C#, and a rebuild on each Jellyfin major.
+
+Status: **v0.1.1.0** — v0.1.0 installed and **verified**: Jellyfin honours the provider's season numbers
+(day seasons appeared on a Shows library). v0.1.1 adds the playback-policy task. Providers, Discord/Authentik
+clients, event history, season-roll + snapshot tasks, Discord feed with frame-grab attachment, config page;
+10 unit tests.
+
+Checklist:
+- [x] skeleton + core season policy (tests) + providers + tasks + feed → `./package.sh` → `dist/VanguardMetadata_0.1.0.0.zip`
+- [ ] Discord dev portal → the existing Authentik application → **Bot** → Reset Token; enable **Server Members**
+      intent; OAuth2 URL generator: scope `bot`, perms View Channels + Send Messages + Embed Links + Attach Files;
+      invite to the server
+- [ ] Authentik: read-only service account + API token for the plugin
+- [ ] install the zip (README in the plugin folder), restart Jellyfin, configure the plugin page
+- [x] libraries → Shows type; providers: only Vanguard; scan — day seasons appeared
+- [ ] install v0.1.1 (`./installJellyfinPlugin.sh`), run "enforce playback policy" once, check a user's policy
+- [ ] verify: seasons appear as "Thursday, Oct 29" / "Week 44" / "October 2026"; poster = avatar;
+      a new file posts to the feed channel with the frame grab
+- [ ] Medal local filename pattern → add to `ItemTimestamp` (need a sample filename)
+- [ ] later: serve `dist/` at `/plugins/` via Caddy as a plugin repo; theme CSS
+
+### Phase 5 — copyparty behind Authentik  (backend, few users; the one that can lock editors out)
+- [ ] Authentik → Providers → Proxy provider, *forward auth (single application)*, external host
+      `https://ovh1.HOST`; Application bound to it; add to the embedded outpost
+- [ ] Caddy: `forward_auth 127.0.0.1:9000 { uri /outpost.goauthentik.io/auth/caddy; copy_headers X-Authentik-Username X-Authentik-Groups … }`
+      on `/`, `/dock*`, `/dump*`, `/sync*`; `reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000`
+- [ ] copyparty: `idp-h-usr: X-Authentik-Username`, `idp-h-grp: X-Authentik-Groups`, `xff-src` = Caddy only;
+      volume accs become `r: @editors`, `rw: @editors` (finals), `rwmd: @admins`
+- [ ] keep the local accounts in `copyparty.conf` until an `editors` Discord user has browsed and
+      downloaded through the new path; then remove them (`configure.py` drops the editors section)
+- [ ] WebDAV: editor creates an app password in Authentik → mounts with Discord username + app password
+- [ ] policy on the Application: only `editors` + `admins` + `streamers` may pass
+
+### Phase 6 — Syncthing UI + dump behind Authentik
+- [ ] `/sync/`: forward-auth, policy `admins` only; Syncthing GUI auth can stay as a second layer
+- [ ] `dumpd.py`: read `X-Authentik-Username` / `X-Authentik-Groups` from Caddy instead of basic auth;
+      admin = member of `admins`; drop `dump-users.txt`
+- [ ] dock: `Dump tonight` sends no credentials; session cookie carries identity
+
+### Phase 7 — dock polish + cleanup
+- [ ] `dumpd.py` gains `GET /me` → `{user, groups, stream_key}` for the logged-in streamer;
+      dock fills OBS settings from it — nobody types a stream password again
+- [ ] `configure.py` shrinks: hostnames + streamers only (editors and admin live in Authentik)
+- [ ] README: replace the per-app login notes with "log in with Discord"
+
+### Phase 8 — hardening  (loose ends from phases 2–3, plus backups)
+- [ ] Jellyfin login branding: Discord button first, "Admin login" reveals the local form
+      (`jellyfin/branding-disclaimer.html` + `jellyfin/branding.css`)
+- [ ] group check: a Discord user in `editors` but not `admins` sees the three libraries and is NOT a
+      Jellyfin administrator; one in `admins` is
+- [x] per-user transcoding OFF for every user, automatically: the Vanguard plugin's "enforce playback
+      policy" task (startup + every 30 min) sets direct-play-only on all non-admin users, SSO-created
+      ones included. Config toggle `EnforceDirectPlayOnly`; turn off on the SYS-3
+- [ ] pin Authentik: `.env` → `AUTHENTIK_TAG=<tag now running>` (`docker inspect authentik-server --format '{{.Config.Image}}'`)
+- [x] guild restriction so only members of our Discord server can log in
+- [ ] rotate the Discord client secret that appeared in a screenshot; paste the new one into the source
+- [ ] nightly `pg_dump` of the Authentik database into `/srv/appdata/authentik/backup/` (timer), keep 14 days;
+      Jellyfin config and Authentik media are plain directories, covered by the archive/backup of `/srv/appdata`
+- [ ] Jellyfin: library "real time monitoring" on, metadata fetchers confirmed off, scheduled scan daily
+
+### Discord bot (separate dev) — runtime provided by the stack
+- `mariadb` container, db `vanguardbot` / user `bot`, loopback only (ssh tunnel for dev access)
+- `discord-bot` container: python 3.12 + ffmpeg + yt-dlp baseline; code bind-mounted from `bot/app/`;
+  env gives token, guild, DB creds, rw `/media/clips`, `/data` scratch; restarts on exit
+- web UI at `bot.` behind Caddy + Authentik forward-auth (first use of the forward-auth pattern that
+  Phase 5 applies to copyparty); the bot trusts `X-Authentik-*` headers and implements no login
+- `bot/AGENT.md` is the build brief for the agent writing the bot; `bot/README.md` the ops cheat-sheet. The clip-submissions feature (Option below) is a natural home
+  for this bot: write `clips/<user>/<name>.mp4` + `<name>.json` sidecar and the Jellyfin plugin picks it up.
+- [ ] CNAME `bot` → main host; `configure.py --full` (asks the hostname, bot token + guild) → `docker compose up -d --build`
+- [ ] Authentik: Proxy Provider (forward auth, single app, external host `https://bot.HOST`) → embedded outpost → group policy
+- [ ] dev drops `main.py` into `bot/app/` → `docker compose restart discord-bot`
+
+### Phase 9 — Gitea + Actions runner
+- `gitea` on MariaDB (db/user from `mariadb/init/10-gitea.sh` on first init, else `./createGiteaDb.sh`),
+  `git.` via Caddy, ssh 2222 (firewall opened), registration closed, Actions enabled
+- `act_runner` with the Docker socket; labels ubuntu-latest / ubuntu-22.04 / dotnet / python
+- [ ] CNAME `git` → main host; `./configure.py --full` (asks the hostname) or `--hosts AUTH WATCH GIT`
+- [ ] https://git.HOST/ install page → admin account
+- [ ] `./registerGiteaRunner.sh`; confirm the runner is idle under Site Administration → Actions → Runners
+- [ ] create users (or OIDC via Authentik, same pattern as Jellyfin: Gitea → Authentication Sources → OpenID Connect)
+- [ ] first use: push `vanguard-stream` itself; a workflow that runs `package.sh` on the `dotnet` label and
+      publishes the plugin zip is the obvious first pipeline
+- risk: the runner's Docker socket is root on the host for anyone who can push a workflow
+
+### Phase 10 — Gitea login via Authentik (OIDC)  (later)
+Same pattern as Jellyfin: Authentik is the identity, Gitea trusts it.
+- [ ] Authentik → Applications → Create with provider → OAuth2/OpenID: name `gitea`, redirect
+      `https://git.HOST/user/oauth2/authentik/callback`, authorization flow implicit-consent, scopes
+      openid email profile (+ `groups` claim via the default profile mapping)
+- [ ] Gitea → Site Administration → Identity & Access → Authentication Sources → Add → OAuth2,
+      provider OpenID Connect, name `authentik` (must match the callback path above),
+      client ID/secret, discovery URL `https://auth.HOST/application/o/<slug>/.well-known/openid-configuration`,
+      "Skip local 2FA", claim mapping: group claim `groups` → Gitea admin group `admins`
+- [ ] Gitea app.ini / env: `GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION=true`,
+      `GITEA__oauth2_client__ACCOUNT_LINKING=auto`, `GITEA__oauth2_client__USERNAME=nickname` so first
+      Discord login creates the Gitea user with no form (registration stays closed for local accounts)
+- [ ] optional: `GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION=true`, hide the password form on the
+      login page (`GITEA__oauth2_client__OPENID_CONNECT_SCOPES`, custom template) so Discord is the only way in
+- [ ] policy on the Authentik application: who may reach Gitea at all (e.g. `admins` + a `devs` group)
+- [ ] test with a non-admin Discord user; confirm admin mapping via `admins` group
+
+### Option — more clip sources (Medal, Twitch/YouTube/Kick clips, Discord uploads)
+
+Today the only clip intake is a local clipper folder synced by Syncthing. Members without that
+setup have no way in. Two additions cover everyone, and neither depends on Authentik.
+
+**A. `#clip-submissions` Discord bot**  (one container, outbound-only, no new ports)
+- member posts a Medal.tv / Twitch clip / YouTube / Kick / Streamable link, or drops a file
+- bot maps the poster's Discord username → `/srv/media/clips/<username>/` (same identity Authentik uses)
+- downloads at best quality with **yt-dlp** (has extractors for all of the above incl. Medal)
+- writes `clip.json` beside it: source URL, title, submitter, timestamp — context for editors
+- reacts ✅ / ❌ on the message; replies with the reason on failure
+- normalize-sweep handles whatever arrives (Medal's HEVC/VFR etc.)
+- config: bot token + channel id in `.env`; yt-dlp pinned, self-updates weekly (extractors rot)
+- quality note: Medal/Twitch/YouTube deliver their re-encode (~1080p). Fine for clips; originals
+  still come via Syncthing or upload below.
+- skipped on purpose: polling Medal profiles via its API — needs per-member IDs, pulls clips nobody
+  submitted, and only returns embed URLs; the bot covers it with explicit intent.
+
+**B. Web upload for members (after Phase 5)**
+- copyparty per-user volume `clips/${u}` writable by its owner, readable by `editors`/`admins`
+- anyone who can log in with Discord drags an original into the browser or WebDAV — full quality,
+  no bot, no install
+
+| source | path in | quality |
+|--------|---------|---------|
+| local clipper + Syncthing | automatic | original |
+| Medal / Twitch / YouTube / Kick link | bot (A) | platform re-encode |
+| Discord attachment | bot (A) | original, Discord size cap |
+| file in hand | web upload (B) | original |
+
+Checklist if adopted:
+- [ ] Discord developer portal → bot user, Message Content intent, invite to the server with
+      read/react/reply in `#clip-submissions`
+- [ ] `bot/` in the bundle: python + discord.py + yt-dlp, runs as uid 1000, mounts `/srv/media/clips`
+- [ ] `.env`: `DISCORD_BOT_TOKEN`, `CLIPS_CHANNEL_ID`
+- [ ] per-user clips volume in `copyparty.conf` once forward-auth is live (Phase 5)
+
+### Later — when it hurts
+- [ ] **SYS-3 (E-2288G)**: Quick Sync → Jellyfin transcoding on, sweep re-encodes in minutes.
+      Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
+- [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
+
+## Risks / gotchas
+
+- **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the
+  Authentik path is proven. Caddy config errors stop *all* sites — `docker logs caddy` first.
+- **Authentik outpost header names** differ by version (`X-Authentik-Username`, `X-Authentik-Groups`,
+  `X-Authentik-Email`). Verify with a test route that echoes headers before wiring copyparty.
+- **Jellyfin SSO plugin** creates users with transcoding allowed by default unless the provider's
+  defaults are set — set them, or the per-user toggle has to be fixed after every first login.
+- **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
+- **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
+
+## Monthly cost
+Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.