Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.
Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
with zero harvesters and no index. Container logs now use filestream + the container parser
with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.
Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
the user holds a privilege and that API does not exist when security is off (the
Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
the only gate. configure.py generates the service passwords, creates the accounts and writes
a root-only netrc for the deadman; --es-security retries just that step.
Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
that the stack's own alerting died.
Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
(loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.
Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with
Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the
first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose
because it is a UI only and its stack is the same piece count with more parts.
configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer
lists actually go away rather than lingering.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.
HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.
LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.
configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.
The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.
check.sh validates all five observability configs; confirmed it catches a deliberate break.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bot's own importer requires an empty destination, but the bot rebuilds part of its state from Discord on
every start, so after the backend switch neither side is a superset. This merges them: log tables append on a
natural key, and the profile/state tables resolve column by column with one explicit rule per column, so the
original joined_at, first_seen_at, rank started_at and tracking start survive a rebuild that stamped today,
while current rank, presence and latest activity stay fresh.
One transaction, idempotent, and it stops rather than guessing when a column has no rule or a table is absent
from the destination. Tested against a fixture built from the real 1041-row file plus a simulated rebuild:
history restored, fresh values kept, 30 membership events (29 historical + tonight's), re-run a no-op, and
both guards verified. A KeyError on insert-only tables during the second run was found and fixed that way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row,
README and the plan doc all describe the installed-release model instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
POST /deploy-bot may now carry X-Vanguard-Env, base64 JSON of the bot's own settings, and the HMAC covers
that header together with the body so the pair cannot be separated or swapped. dumpd merges only the keys
BOT_ENV_KEYS allows (DISCORD_BOT_TOKEN, DISCORD_GUILD_ID by default) into the stack .env, atomically and
preserving mode and owner, and reports the rest to the CI job as settings_refused. The server keeps what it
generates: a CI job cannot touch the database password or the Authentik keys.
Verified: allowed keys applied, server-owned keys refused while the rest still applied, a settings swap
after signing rejected with 401, and an empty settings header leaving .env untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dumpd gains POST /deploy-bot on the bridge-only auth port: HMAC-SHA256 over the body with BOT_DEPLOY_SECRET,
a 300 MB cap, one install at a time, and the installer's output returned in the response so the CI job shows
it. BOT_DEPLOY_AUTO=0 makes the server refuse. configure.py generates the secret, writes both settings into
/etc/vanguard/deploy.env, installs deployBotBundle.sh as /usr/local/bin/vanguard-deploy-bot and creates
/srv/bot/app; --show prints what to paste into Gitea.
Chose this over mounting the Docker socket into job containers, which would give every CI job root on the
host. Also moved threading/subprocess to dumpd's module imports, which the new module-level lock needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.
- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
image now ships none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bin/mtx-on-ready.sh (runOnReady inside the mediamtx container) posts the is-live notice and, for sources whose
audio is not Opus, runs ffmpeg: video copy, audio -> Opus, published over MediaMTX's loopback RTSP to
<user>/<Title>.webrtc (record: no). dumpd accepts .webrtc publishes only from 127.0.0.1. The portal hides
mirror paths, plays the mirror over WHEP (HLS stays on the source), holds an AAC source for 15 s while its
mirror comes up, and corrects the viewer count. Verified headless with a mocked path list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Moves existing panels (video keeps playing) into cols=ceil(sqrt(n)) row-major and equalizes group sizes.
Verified headless for 3-6 streams.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CSS aspect-ratio on a flex:1 child was ignored, so object-fit:cover cropped arbitrarily. Player now
measures the panel (ResizeObserver: fires on every dockview resize/move), fits a frame of the game aspect,
lays the video across its width and hides the overflow at the bottom.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The published recording is the game picture + audio + tmcd track, nothing else. Streams without the
band are still remuxed losslessly. vgstrip plan exports TC_BAND and marks the sidecar band_removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
configure.py: three new host roles (files/sync/clips) in --full, --hosts, .env, Caddy generator;
main host serves about/ at the root (/about* redirects); /time only where it is fetched same-origin.
about/index.html links across hosts; addCommunityMediaServer.sh asks for the new names; Caddyfile.example
regenerated; README/FRESH-INSTALL/HOSTNAME-CHANGE (procedure C)/STATUS updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Jellyfin merges several videos in one non-season folder into a single episode with versions,
which collapsed a night of helmet cams into one entry. Recording still lands in <Title>/.live/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
BOT_REQUIRE_GROUP / LIVE_REQUIRE_GROUP in .env drive a header_regexp matcher after forward_auth
(403 before the app). authentik/discord-roles-to-groups.py syncs Discord roles into groups at
every login so access follows Discord roles without manual edits.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Go Live closes viewer windows, fetches the Authentik-identified user's MediaMTX key from dumpd
(/etc/vanguard/streamers.json written by configure.py), opens Chrome's window-only picker with
audio, prefers H.264 so MediaMTX can record, caps at 12 Mbps / 1440p60, shows live stats, Stop.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
