From 48409b5dbf6de051b92ac5a5bdbc5c403f6323fa Mon Sep 17 00:00:00 2001 From: Brandyn / Techy Date: Sat, 31 Oct 2026 18:00:01 -0500 Subject: [PATCH] Self-service clip intake (/clips + dumpd /me/clips via Syncthing REST) and role-aware gateway page (/about + dumpd /me) Co-Authored-By: Claude Fable 5.1 --- README.md | 16 ++++- STATUS.md | 8 ++- about/index.html | 169 +++++++++++++++++++++++++++++++++++++++++++++ bin/dumpd.py | 114 ++++++++++++++++++++++++++++++ check.sh | 2 +- clips/index.html | 95 +++++++++++++++++++++++++ configure.py | 15 ++++ docker-compose.yml | 2 + 8 files changed, 415 insertions(+), 6 deletions(-) create mode 100644 about/index.html create mode 100644 clips/index.html diff --git a/README.md b/README.md index 0338cfc..fe55a24 100644 --- a/README.md +++ b/README.md @@ -116,9 +116,14 @@ its token expires, but a reconnect needs a fresh one. Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141+ for window audio + GPU encode). -## 5a. Onboarding a gamer's clips (Syncthing) +## 5a. Onboarding a gamer's clips (Syncthing) — self-service -Server side (once per person; name = their Authentik/Discord username, lowercase): +Gamers do it themselves at **https://HOST/clips/** (Discord login): the page shows our Syncthing Device ID and +asks for theirs; on submit, dumpd creates `clips/` with the video-only `.stignore`, adds their device +to our Syncthing (REST API, key read from `/srv/media/.syncthing/config.xml`) and creates the receive-only +folder `clips-` shared with them (min free 300 GB). They accept the device + folder on their PC, +point it at their clip directory, Send Only. The page shows pairing/connection/sync state. Idempotent. +Manual fallback (admin, same result for the server folder): ./addClipUser.sh # creates /srv/media/clips/ with the video-only .stignore @@ -132,6 +137,13 @@ offer pops up: **Add** → Folder Path `/var/syncthing/clips/`, Folder Minimum Free Disk Space **300 GB**, Save. Clips start arriving; the sweep normalizes them; Jellyfin shows the folder as that person's show. Their own folder structure is mirrored; only video files are pulled. +## 5a². Gateway page — https://HOST/about/ + +Role-aware landing page (Discord login): everyone sees Watch / Live / Send clips / Sign out with short how-tos; +`streamers` get the dock + OBS setup steps; `editors` the browser + multicam/timecode notes; `media-admins` +trash/Syncthing/whoami; `git-users` Gitea; `bot-users` the bot; `admins`/`authentik Admins` the admin consoles and +where things live. Identity comes from `GET /me` (dumpd). Point people here first. + ## 5b. Syncthing hardening (do once, in the server's Syncthing UI) Syncthing receives straight into `clips/` (Syncthing UI path `/var/syncthing/clips/`), diff --git a/STATUS.md b/STATUS.md index 9382f09..2b8fff1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -27,7 +27,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea |---|---|---|---| | Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live | | Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://ovh1…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** | -| Gamer | Nothing; clips save where they always did | their clipper + Syncthing, paired once | — | +| Gamer | Self-serve pairing at `/clips/` (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://ovh1…/clips/` + Syncthing | Discord — **pending deploy** | +| Everyone | Landing page with role-aware links and instructions | `https://ovh1…/about/` | Discord — **pending deploy** | | Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; export at the game resolution to drop the band | Resolve / Premiere | — | | Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://ovh1…/` | Discord (Authentik); rights by group `editors` / `media-admins` | | Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) | @@ -36,7 +37,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin | | Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists | | Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh | -| Admin | Pair a gamer's device, approve their folder | Syncthing UI `https://ovh1…/sync/` | Discord + group `media-admins` (Syncthing's own login disabled) | +| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://ovh1…/sync/` | Discord + group `media-admins` (Syncthing's own login disabled) | | Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin | | Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh | | Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh | @@ -47,7 +48,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea |---|---|---| | MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`) | live, token auth verified | | copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live | -| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`; OBS dock with timecode-band setup | identity from Authentik headers | live | +| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity, `/me/clips` Syncthing self-service; OBS dock with timecode-band setup | identity from Authentik headers | live; `/me`, `/me/clips` **pending deploy** | +| `/about/` gateway, `/clips/` self-service | static pages behind the gate | — | **built, pending deploy** | | Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) | | Syncthing | clip intake, UI at `/sync/` | UI behind Authentik + `media-admins`, own GUI auth off | live | | normalize sweep | editing standard, one copy per clip | live | diff --git a/about/index.html b/about/index.html new file mode 100644 index 0000000..cb5aa0e --- /dev/null +++ b/about/index.html @@ -0,0 +1,169 @@ + + + + +Vanguard Media + + + + +
+

Vanguard Media

signed in as …
+
+ +
+

Watch

+

Everything here uses your Discord login. One sign-in covers all of it.

+
+ Recordings & clipsJellyfin — by streamer, by day/week/month + Live nowevery active stream in one page, audio follows the window you click + Send your game clipspair your PC once; clips upload themselves + Sign outends the Discord session for all Vanguard sites +
+
How watching works +
    +
  • Jellyfin: each streamer is a show. This week's recordings sit under weekday seasons, older ones roll into week, month and year. Nothing moves; it is only re-labelled.
    • +
  • Live: windows open as people go live and close when they stop. Click a window to hear it; the speaker button pins audio on; the slider is per window.
    • +
  • Clips: install Syncthing on your gaming PC, paste its Device ID on the clips page, accept the folder it offers. Only video files are pulled.
    • +
+
+
+ + + + + + + + + + + + + + +
+ + + diff --git a/bin/dumpd.py b/bin/dumpd.py index da946b0..86d4d9b 100755 --- a/bin/dumpd.py +++ b/bin/dumpd.py @@ -6,6 +6,9 @@ POST /dump identity from X-Authentik-Username / X-Authentik-Groups (set by Cad GET /dump -> lists what a dump would move (dry run) GET /me/stream-token -> short-lived publish token for the caller (member of streamers/admins); the dock and the live portal set it as the OBS/WHIP credential. No stored passwords anywhere. +GET /me/clips -> Syncthing self-service status for the caller: our device id, their folder/device state +POST /me/clips -> {"device_id": "..."}: create clips/ (+ .stignore), add their device to our Syncthing, + create the receive-only folder clips- shared with them (min free 300 GB). Idempotent. POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret. @@ -90,6 +93,96 @@ def do_move(files, night): d.rmdir() return moved, size +# ------------------------------------------------------------------ Syncthing (self-service clip intake) +import urllib.request, urllib.error +ST_URL = os.environ.get("SYNCTHING_URL", "http://127.0.0.1:8384") +ST_CONFIG = Path(os.environ.get("SYNCTHING_CONFIG", "/srv/media/.syncthing/config.xml")) +ST_CLIPS_PREFIX = os.environ.get("SYNCTHING_CLIPS_PREFIX", "/var/syncthing/clips") # path INSIDE the syncthing container +STIGNORE_SRC = Path(os.environ.get("STIGNORE_SRC", "/etc/vanguard/stignore")) +MIN_FREE_GB = int(os.environ.get("CLIPS_MIN_FREE_GB", "300")) + +def st_apikey(): + try: + m = re.search(r"([^<]+)", ST_CONFIG.read_text()) + return m.group(1) if m else None + except OSError: + return None + +def st(method, path, body=None): + key = st_apikey() + if not key: raise RuntimeError("syncthing api key not found") + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(ST_URL + path, data=data, method=method, headers={"X-API-Key": key, "Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=10) as r: + raw = r.read() + return json.loads(raw) if raw else None + +DEVICE_ID_RE = re.compile(r"^[A-Z2-7]{7}(-[A-Z2-7]{7}){7}$") + +def clips_status(user): + out = {"user": user, "folder_id": f"clips-{user}", "server_path": f"{ROOT}/clips/{user}"} + try: + out["server_device_id"] = st("GET", "/rest/system/status")["myID"] + cfg_devices = st("GET", "/rest/config/devices") + cfg_folders = st("GET", "/rest/config/folders") + folder = next((f for f in cfg_folders if f["id"] == out["folder_id"]), None) + out["folder_configured"] = folder is not None + theirs = [d["deviceID"] for d in (folder or {}).get("devices", []) if d["deviceID"] != out["server_device_id"]] + out["device_id"] = theirs[0] if theirs else None + if out["device_id"]: + conns = st("GET", "/rest/system/connections")["connections"] + out["device_connected"] = bool(conns.get(out["device_id"], {}).get("connected")) + out["device_name"] = next((d["name"] for d in cfg_devices if d["deviceID"] == out["device_id"]), None) + try: + comp = st("GET", f"/rest/db/completion?folder={out['folder_id']}&device={out['device_id']}") + out["remote_completion"] = comp.get("completion") + except Exception: + pass + try: + stt = st("GET", f"/rest/db/status?folder={out['folder_id']}") + out["files_here"] = stt.get("localFiles"); out["state"] = stt.get("state") + except Exception: + pass + except Exception as e: + out["error"] = f"syncthing: {e}" + out["folder_exists"] = Path(out["server_path"]).is_dir() + return out + +def clips_setup(user, device_id, device_name=None): + device_id = device_id.strip().upper() + if not DEVICE_ID_RE.match(device_id): + raise ValueError("that does not look like a Syncthing Device ID (8 groups of 7 characters, e.g. ABCDEFG-HIJKLMN-...)") + # 1. server folder with the video-only ignore list + d = Path(ROOT) / "clips" / user + d.mkdir(parents=True, exist_ok=True); shutil.chown(d, 1000, 1000) + ign = d / ".stignore" + if not ign.exists() and STIGNORE_SRC.exists(): + ign.write_text(STIGNORE_SRC.read_text()); shutil.chown(ign, 1000, 1000); ign.chmod(0o644) + # 2. their device + my_id = st("GET", "/rest/system/status")["myID"] + if device_id == my_id: raise ValueError("that is the server's own Device ID — paste yours (your Syncthing → Actions → Show ID)") + devices = st("GET", "/rest/config/devices") + if not any(x["deviceID"] == device_id for x in devices): + st("POST", "/rest/config/devices", {"deviceID": device_id, "name": device_name or user, "addresses": ["dynamic"], + "autoAcceptFolders": False, "introducer": False, "paused": False}) + # 3. receive-only folder shared with them + fid = f"clips-{user}" + folders = st("GET", "/rest/config/folders") + cur = next((f for f in folders if f["id"] == fid), None) + spec = {"id": fid, "label": f"Clips — {user}", "path": f"{ST_CLIPS_PREFIX}/{user}", "type": "receiveonly", + "devices": [{"deviceID": my_id}, {"deviceID": device_id}], + "minDiskFree": {"value": MIN_FREE_GB, "unit": "GB"}, "fsWatcherEnabled": True, "rescanIntervalS": 3600, + "ignorePerms": True, "ignoreDelete": False} + if cur is None: + st("POST", "/rest/config/folders", spec) + else: + if not any(x["deviceID"] == device_id for x in cur.get("devices", [])): + cur["devices"].append({"deviceID": device_id}) + cur["type"] = "receiveonly"; cur["minDiskFree"] = spec["minDiskFree"] + st("PUT", f"/rest/config/folders/{fid}", cur) + print(f"{datetime.now(TZ):%F %T} clips setup user={user} device={device_id[:7]}… folder={fid}", flush=True) + return clips_status(user) + def token_secret(): try: sec = TOKEN_SECRET_FILE.read_bytes().strip() @@ -164,6 +257,16 @@ class H(BaseHTTPRequestHandler): if self.path.startswith("/time"): # server clock for the timecode strip (the /clock/ page and Go Live measure their offset against this) return self._json(200, {"now_ms": int(_time.time() * 1000)}) + if self.path == "/me" or self.path == "/me/": + who, is_admin = identity(self.headers) + if not who: return self._json(401, {"error": "not authenticated"}) + groups = sorted({g.strip() for g in re.split(r"[|,]", self.headers.get(AUTH_GROUPS_HEADER) or "") if g.strip()}) + return self._json(200, {"user": who, "groups": groups, "admin": is_admin, + "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or ""}) + if self.path.startswith("/me/clips"): + who, _ = identity(self.headers) + if not who: return self._json(401, {"error": "not authenticated"}) + return self._json(200, clips_status(who)) if self.path.startswith("/me/stream-token") or self.path.startswith("/me/stream-key"): # Only reachable through Caddy behind Authentik forward-auth, which injects the identity headers. who, _ = identity(self.headers) @@ -217,6 +320,17 @@ class H(BaseHTTPRequestHandler): return self._json(401, {"error": "path must be /"}) print(f"{datetime.now(TZ):%F %T} mtx-auth allow publish user={user} path={path} ip={req.get('ip')}", flush=True) return self._json(200, {"ok": True}) + if self.path.startswith("/me/clips"): + who, _ = identity(self.headers) + if not who: return self._json(401, {"error": "not authenticated"}) + n = int(self.headers.get("Content-Length") or 0) + try: + body = json.loads(self.rfile.read(n) or b"{}") + return self._json(200, clips_setup(who, str(body.get("device_id", "")), body.get("device_name"))) + except ValueError as e: + return self._json(400, {"error": str(e)}) + except Exception as e: + return self._json(502, {"error": f"syncthing: {e}"}) if not self.path.startswith("/dump"): return self._json(404, {"error": "not found"}) u, is_admin = identity(self.headers) diff --git a/check.sh b/check.sh index 077ad53..5379877 100755 --- a/check.sh +++ b/check.sh @@ -16,7 +16,7 @@ rm -rf __pycache__ bin/__pycache__ python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } node -e " const fs=require('fs'); -for (const f of ['dock/index.html','dock/clock/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } +for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" && say "dock + clock JS" ok || { say "dock + clock JS" "SYNTAX ERROR"; fail=1; } diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null && say "vgclock.js in sync (dock/portal)" ok || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line" # generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse diff --git a/clips/index.html b/clips/index.html new file mode 100644 index 0000000..ab7956c --- /dev/null +++ b/clips/index.html @@ -0,0 +1,95 @@ +<!doctype html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<title>Vanguard Clips + + + + +
+

Send your clips to Vanguard

+
Logged in as …. Your clips land in clips/… and show up in Jellyfin under your name.
+ +
+

Status

+
loading…
+
+
+ +
+

1. Install Syncthing on your gaming PC

+
    +
  1. Windows: Syncthing or Syncthing Tray. Mac/Linux: Syncthing.
    2. +
  2. Open its page (usually http://localhost:8384) → Actions → Show ID. Copy the Device ID.
    3. +
+
+ +
+

2. Paste your Device ID here

+ + +
+
This creates your folder on the server, adds your device, and shares a receive-only folder named clips-… with you.
+
+ +
+

3. Accept on your PC

+
    +
  1. Your Syncthing shows “New Device” from Vanguard → Add Device. Our ID, for checking: + …
    2. +
  2. Then “New Folder” clips-… → Add. Folder Path = the folder your clips save to + (ShadowPlay / Medal / Steam). Folder Type: Send Only. Save.
    3. +
  3. Done. Only video files are pulled; your folder layout is mirrored. The server converts odd formats and + keeps one copy per clip.
    4. +
+
+
+ + + diff --git a/configure.py b/configure.py index 8f9e00a..edc4acd 100755 --- a/configure.py +++ b/configure.py @@ -279,6 +279,20 @@ name: {{http.request.header.X-Authentik-Name}}" file_server }} + # gateway page: role-aware links + instructions (static; talks to /me) + redir /about /about/ 308 + handle_path /about/* {{ + root * /srv/about + file_server + }} + + # clip intake self-service (static page; talks to /me/clips) + redir /clips /clips/ 308 + handle_path /clips/* {{ + root * /srv/clips + file_server + }} + # copyparty. It honours X-Authentik-Username / -Groups only when the request also carries a header whose # secret NAME only this proxy knows (copyparty idp-h-key). Clients cannot forge identity. handle {{ @@ -472,6 +486,7 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis print("== dump users + restart dumpd") run(["sudo", "install", "-d", "-m", "0750", "/etc/vanguard"]) run(["sudo", "install", "-m", "0600", "stream-token.secret", "/etc/vanguard/stream-token.secret"]) + run(["sudo", "install", "-m", "0644", "syncthing/stignore", "/etc/vanguard/stignore"]) # dumpd copies it into new clips folders run(["sudo", "systemctl", "restart", "dumpd"]) print("== containers") dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"] diff --git a/docker-compose.yml b/docker-compose.yml index 0c22b48..187ff54 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -54,6 +54,8 @@ services: volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro - ./dock:/srv/dock:ro + - ./clips:/srv/clips:ro # clip-intake self-service page + - ./about:/srv/about:ro # gateway page (/about) - /srv/live:/srv/live:ro # live portal static build (deployLive.sh) - caddy_data:/data - caddy_config:/config