diff --git a/LOGGING.md b/LOGGING.md index 4eb2b36..f2de9a1 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -38,7 +38,7 @@ into stream labels without parsing a single line. | `media` | sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty | `file` | | `intake` | syncthing (`folder.id`), dumpd `clips_setup`, `quota_*` | `folder` = `clips-` | | `infra` | mariadb, caddy TLS/ACME, container restarts, disk | host only | -| `deploy` | deploy.sh, dumpd `deploy_*` and `bot_deploy_*` (the bot's CI posting a release bundle), act_runner | `rev` | +| `deploy` | deploy.sh, dumpd `deploy_*`, `bot_deploy_*` and `bot_env_*` (the bot's CI posting a release bundle and its settings), act_runner | `rev` | | `code`, `bot` | gitea, discord-bot | — | ## Labels vs fields @@ -55,7 +55,7 @@ into stream labels without parsing a single line. - Same key means the same thing everywhere: `stream` (user/Title), `user`, `file`, `folder`, `rev`, `ip`, `reason`. - Implemented by: `log()` in `bin/dumpd.py`, `vglog` in `bin/normalize-sweep.sh` and `bin/vgdeband.sh`, `log()` in `deploy.sh` (`msg=` only), `bin/mtx-on-ready.sh` (MediaMTX prefixes its own path/time). -Events today: dumpd `publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused`; +Events today: dumpd `publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused bot_env_applied bot_env_failed`; sweep `verdict timecode deband fill_gaps published done failed skip`; vgdeband `deband done failed skip`; hook `mirror`. A future alert script watches `level=error`, `event=failed`, `event=publish_deny` bursts, `event=quota_paused`, plus disk. diff --git a/README.md b/README.md index f7ea4b4..632f908 100644 --- a/README.md +++ b/README.md @@ -253,12 +253,28 @@ HMAC-SHA256 signature over the body, then runs `/usr/local/bin/vanguard-deploy-b (bind-mounted at `/app`, previous release kept beside it) and restarts the container. The install log goes back in the HTTP response, so it appears in the CI job. -Two settings, both generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`: +**The bot's own secrets live in Gitea, not here.** The deploy job also sends the bot's settings, signed with +the bundle, and dumpd writes them into this stack's `.env` — but only the keys `BOT_ENV_KEYS` lists. Anything +else comes back to the CI job as `settings_refused` and is not written, so the database password and the +Authentik keys this repo generates cannot be overwritten from CI. Rotating the Discord token is therefore a +Gitea edit plus a re-run, with no server access. + +In Gitea, on the bot repo under Settings → Actions: | | | |---|---| -| `BOT_DEPLOY_SECRET` | 64 hex chars. Paste it into Gitea → the bot repo → Settings → Actions → Secrets as **`BOT_DEPLOY_SECRET`**. `./configure.py --show` prints it. | -| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 and you install by hand. | +| Secret `BOT_DEPLOY_SECRET` | must equal the server's; `./configure.py --show` prints it | +| Secret `DISCORD_BOT_TOKEN` | the bot token | +| Variable `DISCORD_GUILD_ID` | the server id | +| Variable `VANGUARD_DEPLOY_HOST` | only if the job cannot detect the Docker gateway | + +On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`: + +| | | +|---|---| +| `BOT_DEPLOY_SECRET` | 64 hex chars, the shared signing key | +| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 | +| `BOT_ENV_KEYS` | the only `.env` keys CI may set, `DISCORD_BOT_TOKEN,DISCORD_GUILD_ID` by default. Extend it when the bot needs a new setting. | By hand, any time: diff --git a/STATUS.md b/STATUS.md index efdd990..cb4d32f 100644 --- a/STATUS.md +++ b/STATUS.md @@ -68,7 +68,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback | | act_runner | CI jobs in Docker | **not registered** | | discord-bot | Python runtime, idles until `bot/app/main.py` exists | live, no code yet | -| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI), which verifies it and swaps it into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, the bot token in `.env`, and `BOT_DEPLOY_SECRET` as a Gitea repo secret) | +| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys `BOT_ENV_KEYS` allows into `.env` and swaps the bundle into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, and `BOT_DEPLOY_SECRET` + `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` set in Gitea) | | Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified | ## Unfinished, in rough priority diff --git a/bin/dumpd.py b/bin/dumpd.py index ef82392..d3bf3c0 100755 --- a/bin/dumpd.py +++ b/bin/dumpd.py @@ -14,6 +14,8 @@ POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPL otherwise logs the push and answers 202 "auto-deploy disabled". POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle, HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot. + May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written + into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body. POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret. @@ -262,6 +264,47 @@ def deploy_env(): pass return d +BOT_ENV_DEFAULT_KEYS = "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID" +BOT_ENV_MAX_HEADER = 8192 +_env_write_lock = threading.Lock() + +def stack_env_path(): + """The stack's .env — docker compose interpolates ${...} from it, so the bot's settings live there.""" + repo = deploy_env().get("DEPLOY_REPO_DIR", "") + return (Path(repo) / ".env") if repo else None + +def apply_bot_env(values): + """Merge allowed keys into the stack .env. Returns (applied, refused); never logs a value.""" + allowed = {k.strip() for k in deploy_env().get("BOT_ENV_KEYS", BOT_ENV_DEFAULT_KEYS).split(",") if k.strip()} + refused = sorted(k for k in values if k not in allowed) + wanted = {k: str(v) for k, v in values.items() if k in allowed} + if not wanted: + return [], refused + path = stack_env_path() + if path is None or not path.is_file(): + raise OSError("DEPLOY_REPO_DIR is not set or holds no .env") + bad = [k for k, v in wanted.items() if not re.fullmatch(r"[A-Z][A-Z0-9_]{1,48}", k) or "\n" in v or "\r" in v] + if bad: + raise ValueError(f"not a usable env assignment: {bad}") + with _env_write_lock: + lines = path.read_text().splitlines() + seen = set() + for i, line in enumerate(lines): + if "=" in line and not line.lstrip().startswith("#"): + key = line.split("=", 1)[0].strip() + if key in wanted: + lines[i] = f"{key}={wanted[key]}"; seen.add(key) + for key in wanted: + if key not in seen: lines.append(f"{key}={wanted[key]}") + stat = path.stat() + tmp = path.with_suffix(".env.incoming") + tmp.write_text("\n".join(lines) + "\n") + os.chmod(tmp, stat.st_mode & 0o777); os.chown(tmp, stat.st_uid, stat.st_gid) + os.replace(tmp, path) + applied = sorted(wanted) + log("bot_env_applied", keys=",".join(applied), refused=",".join(refused) or None) + return applied, refused + _bot_deploy_lock = threading.Lock() BOT_DEPLOY_MAX_BYTES = 300 * 1024 * 1024 BOT_DEPLOY_INSTALLER = os.environ.get("BOT_DEPLOY_INSTALLER", "/usr/local/bin/vanguard-deploy-bot") @@ -442,11 +485,26 @@ class H(BaseHTTPRequestHandler): if not 0 < size <= BOT_DEPLOY_MAX_BYTES: return self._json(413, {"error": f"body must be 1..{BOT_DEPLOY_MAX_BYTES} bytes, got {size}"}) raw = self.rfile.read(size) + env_b64 = (self.headers.get("X-Vanguard-Env") or "").strip() + if len(env_b64) > BOT_ENV_MAX_HEADER: + return self._json(413, {"error": "X-Vanguard-Env is too large"}) sig = (self.headers.get("X-Vanguard-Signature") or "").replace("sha256=", "").strip() - if not sig or not hmac.compare_digest(hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest(), sig): + signed = env_b64.encode() + raw # the settings are signed with the bundle, not beside it + if not sig or not hmac.compare_digest(hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest(), sig): log("bot_deploy_bad_signature", level="warn", ip=self.client_address[0], bytes=len(raw)) return self._json(401, {"error": "bad signature"}) + applied, refused = [], [] + if env_b64: + try: + values = json.loads(base64.b64decode(env_b64, validate=True)) + if not isinstance(values, dict): raise ValueError("X-Vanguard-Env must be a JSON object") + applied, refused = apply_bot_env(values) + except (ValueError, OSError, TypeError) as error: + log("bot_env_failed", level="error", error=error) + return self._json(400, {"error": f"settings rejected: {error}"}) code, body = install_bot_bundle(raw, self.headers.get("X-Vanguard-Reason") or "ci") + body["settings_applied"] = applied + if refused: body["settings_refused"] = refused return self._json(code, body) if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/deploy"): # Gitea (container) -> host gateway -> here. Verify HMAC-SHA256 of the raw body with the webhook secret. diff --git a/bot/AGENT.md b/bot/AGENT.md index 059be85..1a4114c 100644 --- a/bot/AGENT.md +++ b/bot/AGENT.md @@ -50,6 +50,12 @@ signed with the repository secret `BOT_DEPLOY_SECRET`; the server verifies the s keeps the previous release for rollback, and restarts the container. The install log comes back in your CI job, so a failed deploy fails the run. Branches and pull requests test and build, and deploy nothing. +**Your settings travel with it.** `DISCORD_BOT_TOKEN` (an Actions secret) and `DISCORD_GUILD_ID` (an Actions +variable) are signed together with the bundle and written into the container's environment by the server, so +this repo is where they are managed and rotated. The server accepts only the names its `BOT_ENV_KEYS` allows +and reports the rest as `settings_refused`; it keeps ownership of what it generates, such as `DB_PASSWORD`. +A new setting means asking the admin to add its name to that list (§6). + Rules that follow from this: - Keep `requirements.txt` resolvable with **prebuilt wheels only**; a dependency that must compile will fail diff --git a/bot/README.md b/bot/README.md index fa5dce8..1bdcc95 100644 --- a/bot/README.md +++ b/bot/README.md @@ -12,6 +12,22 @@ release kept for `--rollback`) and restarts the container. The install log comes A branch or pull request runs the tests and builds the artifact, and deploys nothing. +## Your secrets live in Gitea + +Set them on this repo under Settings → Actions. The deploy job sends them with the bundle, signed, and the +server writes the ones it allows into the environment the container gets: + +| where | name | what | +|---|---|---| +| Secrets | `BOT_DEPLOY_SECRET` | signs the upload; must match the server (the admin has it) | +| Secrets | `DISCORD_BOT_TOKEN` | the bot token | +| Variables | `DISCORD_GUILD_ID` | the Discord server id | + +Rotating the token is an edit here plus a re-run of the workflow; nobody needs to touch the server. The +server keeps what it generates, chiefly `DB_PASSWORD`, and accepts only the names its `BOT_ENV_KEYS` lists — +anything else is reported in the job as `settings_refused`. Need a new setting? Ask the admin to add its name +to that list, then add it here. + The image contains **no** bot dependencies: the container imports yours from `/app/vendor` (`PYTHONPATH=/app/vendor`). Adding a dependency means editing `app/requirements.txt` and pushing — no image rebuild, no package index on the server. The image only changes if you need a new system package (ffmpeg and diff --git a/configure.py b/configure.py index 0936b98..1ce41ba 100755 --- a/configure.py +++ b/configure.py @@ -372,6 +372,7 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b env.setdefault("BOT_DB_PASSWORD", "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32]) env.setdefault("BOT_DEPLOY_SECRET", secrets.token_hex(32)) # Gitea repo secret BOT_DEPLOY_SECRET: signs the bot's release bundle env.setdefault("BOT_DEPLOY_AUTO", "1") # 0 = refuse bundles posted by CI; install by hand instead + env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID") # the only .env keys the bot's CI may set env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot env.setdefault("DISCORD_GUILD_ID", "") env["WATCH_HOST"] = watch_host or "watch.example.com" @@ -537,7 +538,7 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis run(["sudo", "install", "-m", "0600", "stream-token.secret", "/etc/vanguard/stream-token.secret"]) run(["sudo", "install", "-m", "0644", "syncthing/stignore", "/etc/vanguard/stignore"]) # dumpd copies it into new clips folders e = read_env() - Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600) + Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600) run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")): run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"]) @@ -614,6 +615,7 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_ print(f"== mariadb : {domain or 'HOST'}:3306 (public, root local-only) root pass {env.get('MARIADB_ROOT_PASSWORD','(run apply)')}") print(f" bot ci : Gitea -> VanguardBot -> Settings -> Actions -> Secrets: BOT_DEPLOY_SECRET={env.get('BOT_DEPLOY_SECRET','(run apply)')}") print(f" pushes to main install themselves (BOT_DEPLOY_AUTO={env.get('BOT_DEPLOY_AUTO','1')}); by hand: ./deployBotBundle.sh ") + print(f" CI may set these .env keys (BOT_ENV_KEYS): {env.get('BOT_ENV_KEYS','DISCORD_BOT_TOKEN,DISCORD_GUILD_ID')}") print(f" bot : db {env.get('BOT_DB_NAME','vanguardbot')} user {env.get('BOT_DB_USER','bot')} pass {env.get('BOT_DB_PASSWORD','(run apply)')}") print(f" gitea : db gitea user gitea pass {env.get('GITEA_DB_PASSWORD','(run apply)')} (created on MariaDB first init, else ./createGiteaDb.sh)") print("== access : by Authentik group — editors (read streams/clips, write finals), media-admins (everything), admins (dump anyone)")