Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
read, and allows retry/bin only for the admin groups; the path validation is the single thing
between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
them.
Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
verifies each step by reading it back.
Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
binary that is not there is a skip, not a failure.
Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Mentions
- bin/vgnotify.sh is shared by the alert timer and the deadman. A webhook will not ping a role
from <@&id> in the content alone: the request has to whitelist it, so every post now sends an
explicit allowed_mentions. parse is always [] as well, so an @everyone appearing inside a log
line or filename in an alert text can never ping the server.
- ALERT_MENTION is written as role:<id>/user:<id> rather than raw Discord syntax, and
ALERT_MENTION_LEVELS keeps pings to errors by default so warnings arrive quietly. The deadman
pings too, because its alarm means nothing else can get through, but its recovery is info level.
- Both scripts now read /etc/vanguard/alerts.env themselves instead of relying on systemd to
inject it. Running one by hand previously behaved differently from the timer, which is exactly
how "the mention is configured but never appears" happened.
configure.py --set
- Change one setting and apply only what it needs. --full re-prompts for every hostname and
secret, which is far too blunt for pointing the alarms at a new channel, and mistyping a
hostname there has broken TLS before.
- Every Discord credential and webhook is now one commented block at the top of .env, so there is
one place to look, and --show prints the same five together with the secrets truncated.
- Each key applies only its own side effect: the webhook rewrites its root-only file, the deploy
webhook rewrites deploy.env, the bot token recreates the bot container, log retention re-injects
the ILM delete phase. LIVE_WEBHOOK_URL deliberately stops short, because MediaMTX only reads it
at start and a recreate cuts a live recording.
Logout
- The proxy provider pointed at default-provider-invalidation-flow ("Logged out of application"),
which ends only that application's session and leaves the SSO session alive, so /logout dropped
the proxy cookie and forward-auth signed the user straight back in. It looked like a refresh.
default-invalidation-flow ("Logout") is what Authentik's own button uses. --apply now enforces
it had explicitly recommended the wrong flow.
Sweep
- The band-removal branch overwrote $reasocodec> marker, which is the
only signal telling the encode to convert audio. A Go Live stream with both the band and Opus
audio therefore got -c:a copy and shippeachable once Windows Go
Live audio started working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with
Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the
first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose
because it is a UI only and its stack is the same piece count with more parts.
configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer
lists actually go away rather than lingering.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.
HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.
LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty
webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord
notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still
evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in
/etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the
container's user while the repo copy cannot.
alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.
configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.
The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.
check.sh validates all five observability configs; confirmed it catches a deliberate break.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quickest way to confirm the Vanguard profile scope mapping is live, without echoing the raw token: the
browser never sees X-Authentik-Jwt, so printing it would hand the user a bearer token for the proxy provider.
Decoding is display-only and never feeds an authorisation decision. Verified against eight inputs including a
numeric claim, a missing claim, a non-token and malformed base64.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
authentik/vanguard-profile-scope-mapping.py (new, and the first time that mapping is version-controlled rather
than pasted from the docs) emits the Discord snowflake from the account's source connection alongside the
inherited groups, so the bot never has to match a name.
docker-compose.yml: discord-bot moves to its own network. It trusts the identity headers Caddy injects, and it
shared the default bridge with jellyfin, copyparty, syncthing, gitea, act_runner and cadvisor, any of which
could reach :8000 directly and forge them. It needs no sibling container: Caddy uses the published loopback
port and MariaDB is reached through host.docker.internal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bot's own importer requires an empty destination, but the bot rebuilds part of its state from Discord on
every start, so after the backend switch neither side is a superset. This merges them: log tables append on a
natural key, and the profile/state tables resolve column by column with one explicit rule per column, so the
original joined_at, first_seen_at, rank started_at and tracking start survive a rebuild that stamped today,
while current rank, presence and latest activity stay fresh.
One transaction, idempotent, and it stops rather than guessing when a column has no rule or a table is absent
from the destination. Tested against a fixture built from the real 1041-row file plus a simulated rebuild:
history restored, fresh values kept, 30 membership events (29 historical + tonight's), re-run a no-op, and
both guards verified. A KeyError on insert-only tables during the second run was found and fixed that way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row,
README and the plan doc all describe the installed-release model instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh installs deployBotBundle.sh as vanguard-deploy-bot and creates /srv/bot/app; the pipeline
reinstalls it when it changes. FRESH-INSTALL gains section 7a and appendix L (the first release, the
python312 runner label, the Gitea secrets, the Authentik app), with the appendix relettered so it stays in
order. README's rebuild list says the bot's repo is outside this one and what to restore where.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
POST /deploy-bot may now carry X-Vanguard-Env, base64 JSON of the bot's own settings, and the HMAC covers
that header together with the body so the pair cannot be separated or swapped. dumpd merges only the keys
BOT_ENV_KEYS allows (DISCORD_BOT_TOKEN, DISCORD_GUILD_ID by default) into the stack .env, atomically and
preserving mode and owner, and reports the rest to the CI job as settings_refused. The server keeps what it
generates: a CI job cannot touch the database password or the Authentik keys.
Verified: allowed keys applied, server-owned keys refused while the rest still applied, a settings swap
after signing rejected with 401, and an empty settings header leaving .env untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dumpd gains POST /deploy-bot on the bridge-only auth port: HMAC-SHA256 over the body with BOT_DEPLOY_SECRET,
a 300 MB cap, one install at a time, and the installer's output returned in the response so the CI job shows
it. BOT_DEPLOY_AUTO=0 makes the server refuse. configure.py generates the secret, writes both settings into
/etc/vanguard/deploy.env, installs deployBotBundle.sh as /usr/local/bin/vanguard-deploy-bot and creates
/srv/bot/app; --show prints what to paste into Gitea.
Chose this over mounting the Docker socket into job containers, which would give every CI job root on the
host. Also moved threading/subprocess to dumpd's module imports, which the new module-level lock needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.
- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
image now ships none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bin/mtx-on-ready.sh (runOnReady inside the mediamtx container) posts the is-live notice and, for sources whose
audio is not Opus, runs ffmpeg: video copy, audio -> Opus, published over MediaMTX's loopback RTSP to
<user>/<Title>.webrtc (record: no). dumpd accepts .webrtc publishes only from 127.0.0.1. The portal hides
mirror paths, plays the mirror over WHEP (HLS stays on the source), holds an AAC source for 15 s while its
mirror comes up, and corrects the viewer count. Verified headless with a mocked path list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The published recording is the game picture + audio + tmcd track, nothing else. Streams without the
band are still remuxed losslessly. vgstrip plan exports TC_BAND and marks the sidecar band_removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
configure.py: three new host roles (files/sync/clips) in --full, --hosts, .env, Caddy generator;
main host serves about/ at the root (/about* redirects); /time only where it is fetched same-origin.
about/index.html links across hosts; addCommunityMediaServer.sh asks for the new names; Caddyfile.example
regenerated; README/FRESH-INSTALL/HOSTNAME-CHANGE (procedure C)/STATUS updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
