Media-Server-Setup/STATUS.md

# Vanguard stack — what runs, what people do, what's unfinished

Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).

## Runs on its own

| Cadence | What happens | Done by |
|---|---|---|
| Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
| Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
| Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
| Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
| Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
| Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — **pending deploy** |
| Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
| On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | `deploy-pending.timer` — **pending first apply** |
| On push to `main` | (OFF: `DEPLOY_AUTO=0`) Gitea webhook → dumpd → `vanguard-deploy pull` applies only what changed | **built, disabled** |
| Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
| **Not yet running** | Pull of files older than 15 days to the home HDD | `home/rclone-archive.sh`, needs the home box |
| **Not yet running** | Nightly backup of the Authentik database (Postgres) | planned |

## What people do

| Who | Activity | Where | Login |
|---|---|---|---|
| Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | `https://live.…/` | Discord |
| Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
| Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
| Streamer (Windows) | **Go Live** from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as `<user>/HelmetCam-<date>`; no OBS needed | Live portal → Go Live | Discord; key fetched for the logged-in user — **verified end to end** (publish → `.live` → sweep → title folder) |
| Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin |
| Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists |
| Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh |
| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://sync.…/` | Discord + group `media-admins` (Syncthing's own login disabled) |
| Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin |
| Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh |
| Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh |

## Services and their state

| Service | Role | State |
|---|---|---|
| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
| copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
| `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
| Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) |
| Syncthing | clip intake, UI at `sync.` | UI behind Authentik + `media-admins`, own GUI auth off | live |
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
| Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
| MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback |
| act_runner | CI jobs in Docker | **not registered** |
| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys `BOT_ENV_KEYS` allows into `.env` and swaps the bundle into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, and `BOT_DEPLOY_SECRET` + `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` set in Gitea) |
| Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified |

## Unfinished, in rough priority

0a. **WebRTC audio** (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in `bin/mtx-on-ready.sh` + portal mapping. **Deploy pending** (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
0a. **Discord bot**: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the `bot.` host).
   Old SQLite history merged in the same night with `bin/vgbot-merge-sqlite.py` (the bot's own importer refuses a
   non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates
   recovered from the rebuild, live rank and activity kept. Pre-merge dump at `/srv/appdata/discord-bot/before-merge.sql`.
   The tool is idempotent, so re-running it is safe.
0. **Band removal**: deployed 2026-11-01; `vgdeband.sh` catch-up ran on the Vanguard-Training recordings (check `journalctl -u vgdeband` for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with `./deployLive.sh`.

1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.
2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
3. Portal viewer during a real stream: corner chip `webrtc`; Windows Go Live with WarDogs audio (first Windows streamer).
5. Housekeeping: pin `AUTHENTIK_TAG`, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
6. Medal filename pattern in the plugin (need a sample filename).
7. Home archive puller on the home box.
8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
   (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
   from the bot itself now; an Authentik username is matched to one Discord member by name.
9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.

## Verified behaviours worth remembering
- **Never scp generated configs** (`mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt`, `.env`): a template copy wiped all streamers once. They are gitignored now; `*.example.*` are reference only.
- Caddy runs in a container: anything it serves from disk must be a mount (`/srv/live` was missing → 404).
- Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
- Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
- The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
- A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
- MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
- **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
- MediaMTX stamps recording filenames in its container clock: without `TZ` it is UTC and the plugin reads names as Eastern (4 h off). `TZ=America/New_York` is set on the container.
- MediaMTX 1.21 refuses to start without `webrtcAdditionalHosts` or `webrtcIPsFromInterfaces`; configure.py writes the public host in.
- Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
- A stream indexed by Jellyfin while still recording has no duration and skips to the next episode on seek — fixed by recording into `.live/` so nothing sees a file until it is finished and flat.
- Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.