Commit Graph

118 Commits

Author SHA1 Message Date
09882833a0 Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack
Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions
only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four
containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:35:39 -06:00
3662eb3465 Phase 14: diag. host with VictoriaLogs at /logs, ops controls later
Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:33:47 -06:00
12760a94c3 dumpd /me reports the decoded discord_id; the gateway page shows it on hover
The quickest way to confirm the Vanguard profile scope mapping is live, without echoing the raw token: the
browser never sees X-Authentik-Jwt, so printing it would hand the user a bearer token for the proxy provider.
Decoding is display-only and never feeds an authorisation decision. Verified against eight inputs including a
numeric claim, a missing claim, a non-token and malformed base64.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:13:27 -06:00
ba110747b7 Identify the bot's web callers by Discord id, and isolate the bot's network
authentik/vanguard-profile-scope-mapping.py (new, and the first time that mapping is version-controlled rather
than pasted from the docs) emits the Discord snowflake from the account's source connection alongside the
inherited groups, so the bot never has to match a name.

docker-compose.yml: discord-bot moves to its own network. It trusts the identity headers Caddy injects, and it
shared the default bridge with jellyfin, copyparty, syncthing, gitea, act_runner and cadvisor, any of which
could reach :8000 directly and forge them. It needs no sibling container: Caddy uses the published loopback
port and MariaDB is reached through host.docker.internal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 00:13:14 -06:00
31bec6e770 STATUS: the bot's SQLite history is merged into MariaDB
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 00:03:22 -06:00
5c91fa7e99 Add vgbot-merge-sqlite.py: merge the bot's old SQLite history into the live MariaDB
The bot's own importer requires an empty destination, but the bot rebuilds part of its state from Discord on
every start, so after the backend switch neither side is a superset. This merges them: log tables append on a
natural key, and the profile/state tables resolve column by column with one explicit rule per column, so the
original joined_at, first_seen_at, rank started_at and tracking start survive a rebuild that stamped today,
while current rank, presence and latest activity stay fresh.

One transaction, idempotent, and it stops rather than guessing when a column has no rule or a table is absent
from the destination. Tested against a fixture built from the real 1041-row file plus a simulated rebuild:
history restored, fresh values kept, 30 membership events (29 historical + tonight's), re-run a no-op, and
both guards verified. A KeyError on insert-only tables during the second run was found and fixed that way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 00:01:19 -06:00
be12a34f8c Roles policy: log the whole decision, and let one Discord role grant several groups
The table was keyed by role id with a repeated key, so Discord Admins silently mapped to admins only and
never bot-users. It is now role id -> (label, [groups]). With execution logging on, the policy records the
role ids Discord returned, each mapped role and the groups it grants, the roles it ignored, the groups it
withheld, any table group missing from Authentik, and what it added or removed - plus which mechanism the
enrollment branch used, which is the open question on first logins.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 23:34:29 -06:00
f7c96dd8e2 bot docs: CI no longer runs the test suites; they run locally before a push
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 22:44:09 -06:00
ad919cada6 deployBotBundle.sh: fail when the installed release does not serve
A release that crash-loops reported HTTP 200 and a green CI job. The installer now waits up to
BOT_HEALTH_WAIT seconds (45) for the bot to answer on 8000, prints the container log either way, and exits
non-zero if it never does, so dumpd returns 500 and the deploy job goes red. The previous release is left
in place for --rollback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 22:29:51 -06:00
25e9078d9b Drop the last references to the bot's code living in this repo
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row,
README and the plan doc all describe the installed-release model instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:58:34 -06:00
1ee8befdee install.sh, deploy.sh and the runbooks cover the bot bundle path
install.sh installs deployBotBundle.sh as vanguard-deploy-bot and creates /srv/bot/app; the pipeline
reinstalls it when it changes. FRESH-INSTALL gains section 7a and appendix L (the first release, the
python312 runner label, the Gitea secrets, the Authentik app), with the appendix relettered so it stays in
order. README's rebuild list says the bot's repo is outside this one and what to restore where.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:57:52 -06:00
ca10a9895f bot docs: pytest gates the release, the website's node tests are advisory
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:55:41 -06:00
f8136025ce act_runner: add a python312 label (CPython 3.12 + node) for the bot's CI; note that labels are fixed at registration
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:51:57 -06:00
1e4c0e88dc The bot's secrets live in Gitea: signed settings travel with the release bundle
POST /deploy-bot may now carry X-Vanguard-Env, base64 JSON of the bot's own settings, and the HMAC covers
that header together with the body so the pair cannot be separated or swapped. dumpd merges only the keys
BOT_ENV_KEYS allows (DISCORD_BOT_TOKEN, DISCORD_GUILD_ID by default) into the stack .env, atomically and
preserving mode and owner, and reports the rest to the CI job as settings_refused. The server keeps what it
generates: a CI job cannot touch the database password or the Authentik keys.

Verified: allowed keys applied, server-owned keys refused while the rest still applied, a settings swap
after signing rejected with 401, and an empty settings header leaving .env untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:37:35 -06:00
b05f129bed Bot releases install themselves: signed bundle upload to dumpd, no Docker socket in CI
dumpd gains POST /deploy-bot on the bridge-only auth port: HMAC-SHA256 over the body with BOT_DEPLOY_SECRET,
a 300 MB cap, one install at a time, and the installer's output returned in the response so the CI job shows
it. BOT_DEPLOY_AUTO=0 makes the server refuse. configure.py generates the secret, writes both settings into
/etc/vanguard/deploy.env, installs deployBotBundle.sh as /usr/local/bin/vanguard-deploy-bot and creates
/srv/bot/app; --show prints what to paste into Gitea.

Chose this over mounting the Docker socket into job containers, which would give every CI job root on the
host. Also moved threading/subprocess to dumpd's module imports, which the new module-level lock needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:32:34 -06:00
22dcf893d5 deployBotBundle.sh: accept the artifact as Gitea serves it (a zip holding the tar.gz)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:20:04 -06:00
223458550a Bot hosting: releases are self-contained CI bundles, and the bot serves its own web UI
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.

- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
  and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
  WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
  image now ships none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:18:15 -06:00
855651c5a4 Metrics taps: node_exporter + cAdvisor containers, MediaMTX and Caddy metrics on loopback (Prometheus format, store-agnostic)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:55:42 -06:00
704bc8c4d3 Logging framework: logfmt events in our scripts, container labels + rotated json-file logs, Caddy JSON access logs; LOGGING.md
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:53:05 -06:00
88bfda4c9d Sweep/vgdeband: cap x264 at ENCODE_THREADS (2) so conversions stop starving Jellyfin and the live portal
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:30:29 -06:00
f32730ba94 Clips folders: create an empty .stlocal with the folder (Syncthing refuses to scan while the #include target is missing)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:08:58 -06:00
2078a53419 UPCOMING.md: gap list, techy's take, agreed direction (MariaDB metadata, markers, future watch app)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 18:21:38 -06:00
645f447015 Live portal: custom layouts are left alone — new streams chime and wait in a draggable tray; Arrange re-grids and returns to auto
layout store (auto/custom + pending), marked custom on manual move (onDidMovePanel outside arrange), sash drag,
stream/group rename. Tray chips: HTML5 drag accepted via onUnhandledDragOver/onDidDrop (position -> direction),
click = new column. WebAudio two-note chime. Gateway page instructions updated. Verified headless (custom vs auto).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 14:36:01 -06:00
b2e7e9a4bb Gateway page: live-page instructions (layout, audio, rename, group names, maximise); portal: maximise button per tab bar
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 14:17:30 -06:00
d5b2ba502b Live portal: rename streams (tab double-click, persisted per browser) and name tab groups (header slot, session)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 14:12:21 -06:00
9bdf9f1947 WebRTC audio mirror: AAC cannot cross WebRTC, so transcode audio to Opus into <path>.webrtc
bin/mtx-on-ready.sh (runOnReady inside the mediamtx container) posts the is-live notice and, for sources whose
audio is not Opus, runs ffmpeg: video copy, audio -> Opus, published over MediaMTX's loopback RTSP to
<user>/<Title>.webrtc (record: no). dumpd accepts .webrtc publishes only from 127.0.0.1. The portal hides
mirror paths, plays the mirror over WHEP (HLS stays on the source), holds an AAC source for 15 s while its
mirror comes up, and corrects the viewer count. Verified headless with a mocked path list.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:58:15 -06:00
f060277167 Streamer guidance: B-frames 0 (MediaMTX refuses WebRTC playback of H.264 with B-frames)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:29:34 -06:00
3432be691f Live portal: auto-arrange panels as an even grid on join/leave, Arrange button
Moves existing panels (video keeps playing) into cols=ceil(sqrt(n)) row-major and equalizes group sizes.
Verified headless for 3-6 streams.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:25:05 -06:00
68bb0e66d5 Live portal: clip the timecode band with a measured game-aspect frame
CSS aspect-ratio on a flex:1 child was ignored, so object-fit:cover cropped arbitrarily. Player now
measures the panel (ResizeObserver: fires on every dockview resize/move), fits a frame of the game aspect,
lays the video across its width and hides the overflow at the bottom.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:14:25 -06:00
eb36ce771d Last ovh1/about leftovers: check.sh covers the new host roles, procedure A verifies at /, bot README db host
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:11:30 -06:00
0d537ccb65 STATUS: hostname restructure deployed, band removal live
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:09:36 -06:00
2cd936beaf Sweep: crop the timecode band off published streams (re-encode after decode); vgdeband.sh catch-up
The published recording is the game picture + audio + tmcd track, nothing else. Streams without the
band are still remuxed losslessly. vgstrip plan exports TC_BAND and marks the sidecar band_removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 13:05:35 -06:00
bb09565388 Fix README section cross-reference (5e)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:29:33 -06:00
9d67a240d9 Runbook F5b + STATUS: nested-groups scope mapping, verified
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:29:14 -06:00
6128a4ef3a README: nested groups need a second scope mapping (managed profile mapping is direct-only and sticky on proxy providers)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:26:14 -06:00
e885f4c979 deploy.sh: pending mode works without a git checkout (timer no longer fails before the clone switch)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:43:22 -05:00
ca37a3913a configure.py: skip vanguard-deploy install when the pipeline files are not on the box
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:41:18 -05:00
18f4e07de8 Split hosts: media (gateway at /), files, clips, sync; live gains /dock, /dock/clock, /dump, /me, /time
configure.py: three new host roles (files/sync/clips) in --full, --hosts, .env, Caddy generator;
main host serves about/ at the root (/about* redirects); /time only where it is fetched same-origin.
about/index.html links across hosts; addCommunityMediaServer.sh asks for the new names; Caddyfile.example
regenerated; README/FRESH-INSTALL/HOSTNAME-CHANGE (procedure C)/STATUS updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:38:35 -05:00
c1a46d66dc HOSTNAME-CHANGE.md: hostname inventory, subdomain rename and base-domain move procedures; linked from README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:23:33 -05:00
3c407c0733 FRESH-INSTALL: appendix runbook with every action and command, A–L
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:21:34 -05:00
861a88fc7e FRESH-INSTALL.md: ordered walkthrough from empty box to working pipeline, manual steps included; linked from README
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:19:06 -05:00
49d33e8210 README: rollout block matches the current one (compose included, visible apply output)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:16:42 -05:00
abb7186bd0 Rebuild gaps closed: Authentik bootstrap creds, Gitea install lock + admin user from .env, Syncthing GUI-auth-off fixup, plugin ensured on apply, portal built by install.sh; README rebuild section with the honest manual remainder
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:14:46 -05:00
33482bfff3 README: push-to-deploy rollout commands, clone switch, webhook test, enabling steps
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:09:42 -05:00
66f05cf9a7 Push-to-deploy (OFF by default): vanguard-deploy applies only changed parts with a live-stream guard for MediaMTX, rollback, pending timer; dumpd /deploy webhook gated by DEPLOY_AUTO; configure --apply; switchToClone.sh; draft CI workflow; committed plugin zip
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:08:43 -05:00
07ffae0746 Soft per-user clip quotas: dumpd pauses/resumes Syncthing folders by folder size (CLIPS_QUOTA_GB, per-user overrides file); clips page shows usage
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 18:30:22 -05:00
c56a2997ba about: clips section grouped with contributing, after Stream
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 18:26:17 -05:00
272791ea7d about: dedicated 'Send your game clips' section for everyone
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 18:25:56 -05:00
2d8e6bb68c about: show only sections the account's groups actually permit
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 18:24:33 -05:00
4b2f3175ad about: section leads state actual membership; admin view warns where a gate will still refuse
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 18:21:00 -05:00