Media-Server-Setup/STATUS.md

Vanguard stack — what runs, what people do, what's unfinished

Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03. Box: OVH SYS-1, Debian 13, ssh ovh1.dumpstervigilantes.rip. Web hosts (one Discord login): media. gateway at /, files. copyparty, clips., sync., live. (portal + /dock/, /dump, /time), watch., git., bot., diag. (logs/metrics/alerts), auth. — all under dumpstervigilantes.rip. Restructure deployed 2026-11-01 (ovh1 is ssh-only now). Bundle: ~/vanguard-stream on the box. Source of truth: this repo, techy/Media-Server-Setup on the stack's own Gitea (ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git).

Runs on its own

Cadence	What happens	Done by
Continuous	OBS streams record into a hidden streams/<user>/<Title>/.live/ folder, invisible to Jellyfin and copyparty until finished	MediaMTX
Continuous	Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only)	Syncthing
Continuous	New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab	Jellyfin plugin (needs bot token + channel in its config)
Every 5 min	Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of <Title>/.live/ to <user>/<Title>__<ts>.mp4 with a frame-accurate tmcd track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip	normalize sweep (systemd timer)
Every 5 min	Discord scheduled events snapshotted into a history so recordings can be matched to their session	Jellyfin plugin
Every 30 min + startup	Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box)	Jellyfin plugin
Daily 04:10 ET	Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change	Jellyfin plugin
Every 5 min	Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage	dumpd quota loop — pending deploy
Daily 05:00 ET	Dumped nights older than 3 days deleted for good	trash purge (systemd timer)
On boot	Everything restarts itself after the media mount is up; interrupted recordings stay playable	systemd + Docker
Every 5 min	If a deploy deferred a MediaMTX restart, retry when no stream is live	deploy-pending.timer — pending first apply
On push to main	(OFF: DEPLOY_AUTO=0) Gitea webhook → dumpd → vanguard-deploy pull applies only what changed	built, disabled
Continuous	Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts	MediaMTX + live portal (page deployed; MediaMTX port fix pending apply)
Not yet running	Pull of files older than 15 days to the home HDD	home/rclone-archive.sh, needs the home box
Not yet running	Nightly backup of the Authentik database (Postgres)	planned

What people do

Who	Activity	Where	Login
Streamer	One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode	OBS dock / Go Live	live
Streamer	Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token	OBS browser dock https://live.…/dock/	Discord login in the dock; dock mints a 12 h token per Go Live — no passwords, verified live
Gamer	Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did	https://clips.…/ + Syncthing	Discord (any server member)
Viewer	Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto	https://live.…/	Discord
Everyone	Landing page with role-aware links and instructions	https://media.…/	Discord
Editor	Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep)	Resolve / Premiere	—
Editor	Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals	copyparty https://files.…/	Discord (Authentik); rights by group editors / media-admins
Anyone in the Discord server	Watch recordings and clips by streamer and season	Jellyfin https://watch.…/	Discord (Authentik)
Anyone in the Discord server	Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume	Live portal https://live.…/	Discord (Authentik, domain-level) — live, shows a real stream
Streamer (Windows)	Go Live from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as <user>/HelmetCam-<date>; no OBS needed	Live portal → Go Live	Discord; key fetched for the logged-in user — verified end to end (publish → .live → sweep → title folder)
Devs	Host code, review, CI	Gitea https://git.…/, ssh 2222	Discord (Authentik), group git-users; git-admins = site admin
Bot dev	Push code, restart container, read logs; web UI at https://bot.…/	Gitea + compose; bot/AGENT.md	web UI behind Discord (Authentik), once its provider exists
Admin	Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups	./configure.py	ssh
Admin	Watch pairings / folder states (gamers self-serve the setup)	Syncthing UI https://sync.…/	Discord + group media-admins (Syncthing's own login disabled)
Admin	Groups, who may log in, app access	Authentik https://auth.…/	Discord or akadmin
Admin	Install/upgrade the Jellyfin plugin	./installJellyfinPlugin.sh	ssh
Admin	Register the CI runner	./registerGiteaRunner.sh	ssh

Services and their state

Service	Role	State
MediaMTX	RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via /mtx-auth); on-ready hook posts "is live" and runs the WebRTC audio mirror (<path>.webrtc, audio → Opus, never recorded)	live, token auth verified
copyparty	browse/scrub/download/finals upload	behind Authentik, group permissions (editors, media-admins), Discord-only, no WebDAV
dumpd + dock	dump, stream tokens, MediaMTX auth hook, /time, /me identity (on live.), /me/clips Syncthing self-service (on clips.); OBS dock with timecode-band setup	identity from Authentik headers
media. gateway, clips. self-service	static pages behind the gate	—
Timecode	bin/vgstrip.py decoder, tmcd + sidecar in the sweep, /dock/clock/ strip page, Go Live band composite, portal crop	server clock is the reference
Syncthing	clip intake, UI at sync.	UI behind Authentik + media-admins, own GUI auth off
normalize sweep	editing standard, one copy per clip	live
nftables	host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000)	live
Caddy	TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped)	live (access logs: deploy pending)
Observability (Phase 13)	Elastic: Elasticsearch (logs + metrics in one store), Kibana = the whole diag. host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics	deployed 2026-11-03. The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 removed the container input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: ALERT_WEBHOOK_URL, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector)
Alerting	Kibana evaluates, we deliver. A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the vanguard-alerts index through the free .index connector and vanguard-alerts.timer takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; observability/kibana-setup.py creates the connector and the heartbeat rule	added 2026-11-03
Alarm rules	Six, created by observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears	added 2026-11-03; needs the bot release that fixes /health before the bot-unhealthy rule means anything
Deadman	vanguard-deadman.timer on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, each Beat actually writing (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set	live 2026-11-03 — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy
Our own metrics	bin/vgmetrics.sh counters from the sweep and vgdeband + live gauges from dumpd GET /metrics (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the .webrtc mirrors separated. vanguard_media_pending trending up across a night is the measured signal that CPU-only encoding is not keeping pace	added 2026-11-03
Logging framework	rotated Docker logs + vanguard.service/category labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (LOGGING.md). Consumed by Filebeat into Elasticsearch	live (labels + logfmt); Elastic deploy pending
Authentik (+Postgres, Redis)	Discord login, groups, guild-membership policy	live
Jellyfin	watch page, SSO via Authentik, direct play only	live
Vanguard Metadata plugin	seasons by tag, Discord names/avatars/events, feed, playback policy	0.1.3 built (flat <Title>__<ts> names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2
MariaDB	databases for the bot and Gitea; public :3306, root local-only	live
Gitea	git hosting on MariaDB, Actions enabled; holds this repo	live; Discord login via Authentik OIDC verified; access = git-users/git-admins groups; local admin as fallback
act_runner	CI jobs in Docker	not registered
bot. host + Discord bot	the bot's API and its web pages on one port (WEB_SERVE_UI), behind Authentik forward-auth; Caddy requires bot-users, the bot re-checks bot-users,admins per request	a push to main deploys itself: CI builds a release bundle (app + vendor/ dependencies) and posts it signed to dumpd POST /deploy-bot (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys BOT_ENV_KEYS allows into .env and swaps the bundle into /srv/bot/app → /app; image installs no bot dependencies
Live portal (live/, live. host)	React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd /me/stream-key	live; Go Live verified

Unfinished, in rough priority

0a. WebRTC audio (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in bin/mtx-on-ready.sh + portal mapping. Deploy pending (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback. 0a. Discord bot: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the bot. host). Old SQLite history merged in the same night with bin/vgbot-merge-sqlite.py (the bot's own importer refuses a non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates recovered from the rebuild, live rank and activity kept. Pre-merge dump at /srv/appdata/discord-bot/before-merge.sql. The tool is idempotent, so re-running it is safe. 0. Band removal: deployed 2026-11-01; vgdeband.sh catch-up ran on the Vanguard-Training recordings (check journalctl -u vgdeband for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with ./deployLive.sh.

1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (/whoami should show bot-users). Nested groups verified 2026-11-01: git-admins now reaches the header via the Vanguard profile (nested groups) scope mapping (README 5e). akadmin remains the recovery login if a policy misfires.
2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
3. Portal viewer during a real stream: corner chip webrtc; Windows Go Live with WarDogs audio (first Windows streamer).
4. Housekeeping: pin AUTHENTIK_TAG, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
5. Medal filename pattern in the plugin (need a sample filename).
6. Home archive puller on the home box.
7. Plugin repository served by Caddy so Jellyfin updates from its catalog.
8. Discord bot: first bundle install (./deployBotBundle.sh), Authentik application for bot., first real MariaDB run (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served from the bot itself now; an Authentik username is matched to one Discord member by name. 9a. Kibana first-run: a data view over filebeat-*,metricbeat-*, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new vanguard_media_* metrics. 9b. Elasticsearch security as its own phase, if diag. is ever to widen past media-admins: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected Authorization). Per-person identity is reachable but buys little, since audit logging is itself paid.
9. Authentik blueprint from .env (Discord source, groups, policies, provider) for reproducible installs.
10. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.

Verified behaviours worth remembering

• Never scp generated configs (mediamtx.yml, copyparty.conf, Caddyfile, dump-users.txt, .env): a template copy wiped all streamers once. They are gitignored now; *.example.* are reference only.
• Caddy runs in a container: anything it serves from disk must be a mount (/srv/live was missing → 404).
• Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
• Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as <Title>__<ts>.mp4 in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
• Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. Gap filling is available (FILL_GAPS_MS in normalize-sweep.service, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
• The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
• An encode that fails with Error opening output files: Invalid argument is usually contradictory flags, not the path. ffmpeg rejects -r together with a non-CFR -fps_mode; the real message is one line earlier in its stderr, which the sweep now captures into the failure log. Found 2026-11-03 on the 1 November test recordings, which also exposed that avg_frame_rate can be pure fiction (95 fps reported for a 30 fps source) — the sweep now takes the lower of r_frame_rate and avg_frame_rate.
• A deploy block that is only half-run leaves old binaries in /usr/local/bin; grep -c the installed script before blaming logic.
• MediaMTX does not reload mediamtx.yml live here (bind-mounted file): any change to it needs docker compose restart mediamtx between streams. Nothing day-to-day touches that file.
• Timecode verified live (2026-10-31): OBS + NVENC with the band → source: strip, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
• MediaMTX stamps recording filenames in its container clock: without TZ it is UTC and the plugin reads names as Eastern (4 h off). TZ=America/New_York is set on the container.
• MediaMTX 1.21 refuses to start without webrtcAdditionalHosts or webrtcIPsFromInterfaces; configure.py writes the public host in.
• Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
• A stream indexed by Jellyfin while still recording has no duration and skips to the next episode on seek — fixed by recording into .live/ so nothing sees a file until it is finished and flat.
• Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
• Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
• Never systemctl stop|restart nftables (flushes Docker's NAT). Reload only.
• Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a tmcd data stream is the fingerprint. First working Windows audio: 2026-11-03.
• The sweep's band-removal branch overwrote $reason, which carried the probe's audio=<codec> marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got -c:a copy and shipped Opus inside an MP4. Fixed by keeping probe_reason separately; it only became reachable once Go Live audio worked.
• Authentik has two invalidation flows and the names are a trap. default-provider-invalidation-flow is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so /logout drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. default-invalidation-flow ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.
• Prefer changing a proxy provider through the ORM (ak shell) over the UI: saving it in the UI re-adds the managed property mappings every time.
• Filebeat 9 removed the container input (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need filestream + parsers: [container: ~] and a stable id each. add_docker_metadata also needs match_fields: ["container.id"] here, or there are no container.labels.* and the service.name/vanguard.category promotion yields empty fields.
• A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set setup.template.settings.index.number_of_replicas: 0.
• VAR=value . file does not work in bash: assignments prefixed to the . builtin are discarded when source returns, so the sourced file's own ${VAR:-default} writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
• find ... | while read runs the loop in a subshell, so anything it accumulates is lost at the end of the pipe. normalize-sweep.sh reads from a process substitution for exactly this reason.
• Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With xpack.security.enabled: false every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on diag.. Anything in Kibana that asks Elasticsearch for privileges fails with no handler found for uri [/_security/user/_has_privileges]. The Infrastructure / Metrics app does this on every page view, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over metricbeat-* instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.