Brandyn / Techy
6f77764fec
addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild sets it up like every other host instead of leaving it to be discovered missing. HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply, and the plan's risks section carries the same warning plus the path-prefix cascade. LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than half-configured; the old Afterwards section is relettered N. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
17 KiB
17 KiB
Vanguard stack — what runs, what people do, what's unfinished
Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01.
Box: OVH SYS-1, Debian 13, ssh ovh1.dumpstervigilantes.rip. Web hosts (one Discord login): media. gateway at /, files. copyparty, clips., sync., live. (portal + /dock/, /dump, /time), watch., git., bot., diag. (logs/metrics/alerts), auth. — all under dumpstervigilantes.rip. Restructure deployed 2026-11-01 (ovh1 is ssh-only now). Bundle: ~/vanguard-stream on the box.
Source of truth: this repo, techy/Media-Server-Setup on the stack's own Gitea (ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git).
Runs on its own
| Cadence | What happens | Done by |
|---|---|---|
| Continuous | OBS streams record into a hidden streams/<user>/<Title>/.live/ folder, invisible to Jellyfin and copyparty until finished |
MediaMTX |
| Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
| Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of <Title>/.live/ to <user>/<Title>__<ts>.mp4 with a frame-accurate tmcd track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip |
normalize sweep (systemd timer) |
| Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
| Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
| Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — pending deploy |
| Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
| On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | deploy-pending.timer — pending first apply |
On push to main |
(OFF: DEPLOY_AUTO=0) Gitea webhook → dumpd → vanguard-deploy pull applies only what changed |
built, disabled |
| Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
| Not yet running | Pull of files older than 15 days to the home HDD | home/rclone-archive.sh, needs the home box |
| Not yet running | Nightly backup of the Authentik database (Postgres) | planned |
What people do
| Who | Activity | Where | Login |
|---|---|---|---|
| Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock https://live.…/dock/ |
Discord login in the dock; dock mints a 12 h token per Go Live — no passwords, verified live |
| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | https://clips.…/ + Syncthing |
Discord (any server member) |
| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | https://live.…/ |
Discord |
| Everyone | Landing page with role-aware links and instructions | https://media.…/ |
Discord |
| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty https://files.…/ |
Discord (Authentik); rights by group editors / media-admins |
| Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin https://watch.…/ |
Discord (Authentik) |
| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal https://live.…/ |
Discord (Authentik, domain-level) — live, shows a real stream |
| Streamer (Windows) | Go Live from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as <user>/HelmetCam-<date>; no OBS needed |
Live portal → Go Live | Discord; key fetched for the logged-in user — verified end to end (publish → .live → sweep → title folder) |
| Devs | Host code, review, CI | Gitea https://git.…/, ssh 2222 |
Discord (Authentik), group git-users; git-admins = site admin |
| Bot dev | Push code, restart container, read logs; web UI at https://bot.…/ |
Gitea + compose; bot/AGENT.md |
web UI behind Discord (Authentik), once its provider exists |
| Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | ./configure.py |
ssh |
| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI https://sync.…/ |
Discord + group media-admins (Syncthing's own login disabled) |
| Admin | Groups, who may log in, app access | Authentik https://auth.…/ |
Discord or akadmin |
| Admin | Install/upgrade the Jellyfin plugin | ./installJellyfinPlugin.sh |
ssh |
| Admin | Register the CI runner | ./registerGiteaRunner.sh |
ssh |
Services and their state
| Service | Role | State |
|---|---|---|
| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via /mtx-auth); on-ready hook posts "is live" and runs the WebRTC audio mirror (<path>.webrtc, audio → Opus, never recorded) |
live, token auth verified |
| copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (editors, media-admins), Discord-only, no WebDAV |
| dumpd + dock | dump, stream tokens, MediaMTX auth hook, /time, /me identity (on live.), /me/clips Syncthing self-service (on clips.); OBS dock with timecode-band setup |
identity from Authentik headers |
media. gateway, clips. self-service |
static pages behind the gate | — |
| Timecode | bin/vgstrip.py decoder, tmcd + sidecar in the sweep, /dock/clock/ strip page, Go Live band composite, portal crop |
server clock is the reference |
| Syncthing | clip intake, UI at sync. |
UI behind Authentik + media-admins, own GUI auth off |
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind diag. (media-admins) |
deploying 2026-11-03: containers up; four first-run corrections applied (systemd 257 rejects current_boot_only: false; -http.pathPrefix moves the ingestion path too; each proxied UI needs its own prefix; a Caddy respond body eats {...}). diag. needed --hosts, not --apply — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No ALERT_WEBHOOK_URL yet, which is a supported state: alerts evaluate and show at /alerts but are not delivered |
| Deadman | vanguard-deadman.timer on the host, every 5 min, curl only: all six components answering, the Watchdog heartbeat through vmalert → Alertmanager → dumpd /heartbeat, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set |
committed 2026-11-03, deploy pending |
| Logging framework | rotated Docker logs + vanguard.service/category labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (LOGGING.md); store not chosen (Loki vs VictoriaLogs vs Netdata) |
committed 2026-11-01, deploy pending (recreates every container) |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
| Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | 0.1.3 built (flat <Title>__<ts> names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
| MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC verified; access = git-users/git-admins groups; local admin as fallback |
| act_runner | CI jobs in Docker | not registered |
bot. host + Discord bot |
the bot's API and its web pages on one port (WEB_SERVE_UI), behind Authentik forward-auth; Caddy requires bot-users, the bot re-checks bot-users,admins per request |
a push to main deploys itself: CI builds a release bundle (app + vendor/ dependencies) and posts it signed to dumpd POST /deploy-bot (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys BOT_ENV_KEYS allows into .env and swaps the bundle into /srv/bot/app → /app; image installs no bot dependencies |
Live portal (live/, live. host) |
React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd /me/stream-key |
live; Go Live verified |
Unfinished, in rough priority
0a. WebRTC audio (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in bin/mtx-on-ready.sh + portal mapping. Deploy pending (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
0a. Discord bot: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the bot. host).
Old SQLite history merged in the same night with bin/vgbot-merge-sqlite.py (the bot's own importer refuses a
non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates
recovered from the rebuild, live rank and activity kept. Pre-merge dump at /srv/appdata/discord-bot/before-merge.sql.
The tool is idempotent, so re-running it is safe.
0. Band removal: deployed 2026-11-01; vgdeband.sh catch-up ran on the Vanguard-Training recordings (check journalctl -u vgdeband for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with ./deployLive.sh.
- Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (
/whoamishould showbot-users). Nested groups verified 2026-11-01:git-adminsnow reaches the header via theVanguard profile (nested groups)scope mapping (README 5e).akadminremains the recovery login if a policy misfires. - Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
- Portal viewer during a real stream: corner chip
webrtc; Windows Go Live with WarDogs audio (first Windows streamer). - Housekeeping: pin
AUTHENTIK_TAG, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning. - Medal filename pattern in the plugin (need a sample filename).
- Home archive puller on the home box.
- Plugin repository served by Caddy so Jellyfin updates from its catalog.
- Discord bot: first bundle install (
./deployBotBundle.sh), Authentik application forbot., first real MariaDB run (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served from the bot itself now; an Authentik username is matched to one Discord member by name. - Authentik blueprint from
.env(Discord source, groups, policies, provider) for reproducible installs. - SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
Verified behaviours worth remembering
- Never scp generated configs (
mediamtx.yml,copyparty.conf,Caddyfile,dump-users.txt,.env): a template copy wiped all streamers once. They are gitignored now;*.example.*are reference only. - Caddy runs in a container: anything it serves from disk must be a mount (
/srv/livewas missing → 404). - Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
- Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as
<Title>__<ts>.mp4in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode. - Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. Gap filling is available (
FILL_GAPS_MSinnormalize-sweep.service, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR. - The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
- A deploy block that is only half-run leaves old binaries in
/usr/local/bin;grep -cthe installed script before blaming logic. - MediaMTX does not reload
mediamtx.ymllive here (bind-mounted file): any change to it needsdocker compose restart mediamtxbetween streams. Nothing day-to-day touches that file. - Timecode verified live (2026-10-31): OBS + NVENC with the band →
source: strip, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path. - MediaMTX stamps recording filenames in its container clock: without
TZit is UTC and the plugin reads names as Eastern (4 h off).TZ=America/New_Yorkis set on the container. - MediaMTX 1.21 refuses to start without
webrtcAdditionalHostsorwebrtcIPsFromInterfaces; configure.py writes the public host in. - Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
- A stream indexed by Jellyfin while still recording has no duration and skips to the next episode on seek — fixed by recording into
.live/so nothing sees a file until it is finished and flat. - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never
systemctl stop|restart nftables(flushes Docker's NAT). Reload only.