Both Beats already wrote to the same Elasticsearch, but a metric event carried container.name while a log
event carried service.name and vanguard.category, so nothing joined them and a dashboard could only show the
two side by side. Metricbeat now runs add_docker_metadata and the same copy_fields, giving both sides
@timestamp, host.name, container.name, service.name and vanguard.category.
LOGGING.md documents the join keys and the one Kibana step that makes it work: a single data view over
filebeat-*,metricbeat-* so one filter applies to metric panels and log panels together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for
Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers
host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and
Caddy's Prometheus endpoints.
Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box
whose spare capacity belongs to x264), single-node discovery, security off because it only listens on
loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start
without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart.
diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links.
The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is
not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when
space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually
written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the
disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.
HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.
LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VictoriaLogs' -http.pathPrefix applies to every endpoint, not just the UI, so the elasticsearch bulk path is
/logs/insert/elasticsearch/. It rejected the writes with exactly that message. The other three references
(vmalert datasource, the metrics scrape, the deadman health probe) were already prefixed correctly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--apply is stateless by design: it reads the hostnames back out of the Caddyfile. A role that is not in there
yet therefore cannot be written by it, which is why the deploy brought every container up and left diag with
no site block and no certificate. --hosts now takes DIAG as its ninth argument and its usage text explains the
constraint instead of leaving it to be rediscovered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
systemd 250-257 refuse to read past boots through journalctl, which made the source config invalid and put
the container in a restart loop with exit 78. The object!() casts were only warnings: inside the err == null
branch VRL has already narrowed the parsed value, so get() takes it directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without those three directories Docker creates them root-owned and victorialogs, victoriametrics (uid 1000)
and alertmanager (65534) cannot write, so the apply has to make them the way it already makes /srv/bot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each service now runs behind the prefix it is published under (-http.pathPrefix, or route-prefix plus
external-url for Alertmanager). Without that, three of the four UIs would have loaded at their sub-path and
then fetched their assets from the domain root, so the pages would have come up broken. The prefixes cascade,
so the scrape paths, vmalert's notifier and remote-write URLs, and the deadman's health probes all follow.
The index is diag/index.html mounted into Caddy, not an inline respond: Caddy reads {...} as placeholders and
the page's CSS is full of braces, so inline would have been mangled. It also matches how the gateway, dock and
clips pages are already served.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty
webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord
notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still
evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in
/etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the
container's user while the repo copy cannot.
alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.
configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.
The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.
check.sh validates all five observability configs; confirmed it catches a deliberate break.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six containers, all loopback-published, reachable only through the gated diag. host:
victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
victoriametrics 90d, scrapes the four taps that were already live plus the new services
vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config
The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.
Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions
only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four
containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quickest way to confirm the Vanguard profile scope mapping is live, without echoing the raw token: the
browser never sees X-Authentik-Jwt, so printing it would hand the user a bearer token for the proxy provider.
Decoding is display-only and never feeds an authorisation decision. Verified against eight inputs including a
numeric claim, a missing claim, a non-token and malformed base64.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
authentik/vanguard-profile-scope-mapping.py (new, and the first time that mapping is version-controlled rather
than pasted from the docs) emits the Discord snowflake from the account's source connection alongside the
inherited groups, so the bot never has to match a name.
docker-compose.yml: discord-bot moves to its own network. It trusts the identity headers Caddy injects, and it
shared the default bridge with jellyfin, copyparty, syncthing, gitea, act_runner and cadvisor, any of which
could reach :8000 directly and forge them. It needs no sibling container: Caddy uses the published loopback
port and MariaDB is reached through host.docker.internal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bot's own importer requires an empty destination, but the bot rebuilds part of its state from Discord on
every start, so after the backend switch neither side is a superset. This merges them: log tables append on a
natural key, and the profile/state tables resolve column by column with one explicit rule per column, so the
original joined_at, first_seen_at, rank started_at and tracking start survive a rebuild that stamped today,
while current rank, presence and latest activity stay fresh.
One transaction, idempotent, and it stops rather than guessing when a column has no rule or a table is absent
from the destination. Tested against a fixture built from the real 1041-row file plus a simulated rebuild:
history restored, fresh values kept, 30 membership events (29 historical + tonight's), re-run a no-op, and
both guards verified. A KeyError on insert-only tables during the second run was found and fixed that way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The table was keyed by role id with a repeated key, so Discord Admins silently mapped to admins only and
never bot-users. It is now role id -> (label, [groups]). With execution logging on, the policy records the
role ids Discord returned, each mapped role and the groups it grants, the roles it ignored, the groups it
withheld, any table group missing from Authentik, and what it added or removed - plus which mechanism the
enrollment branch used, which is the open question on first logins.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A release that crash-loops reported HTTP 200 and a green CI job. The installer now waits up to
BOT_HEALTH_WAIT seconds (45) for the bot to answer on 8000, prints the container log either way, and exits
non-zero if it never does, so dumpd returns 500 and the deploy job goes red. The previous release is left
in place for --rollback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row,
README and the plan doc all describe the installed-release model instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh installs deployBotBundle.sh as vanguard-deploy-bot and creates /srv/bot/app; the pipeline
reinstalls it when it changes. FRESH-INSTALL gains section 7a and appendix L (the first release, the
python312 runner label, the Gitea secrets, the Authentik app), with the appendix relettered so it stays in
order. README's rebuild list says the bot's repo is outside this one and what to restore where.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
POST /deploy-bot may now carry X-Vanguard-Env, base64 JSON of the bot's own settings, and the HMAC covers
that header together with the body so the pair cannot be separated or swapped. dumpd merges only the keys
BOT_ENV_KEYS allows (DISCORD_BOT_TOKEN, DISCORD_GUILD_ID by default) into the stack .env, atomically and
preserving mode and owner, and reports the rest to the CI job as settings_refused. The server keeps what it
generates: a CI job cannot touch the database password or the Authentik keys.
Verified: allowed keys applied, server-owned keys refused while the rest still applied, a settings swap
after signing rejected with 401, and an empty settings header leaving .env untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dumpd gains POST /deploy-bot on the bridge-only auth port: HMAC-SHA256 over the body with BOT_DEPLOY_SECRET,
a 300 MB cap, one install at a time, and the installer's output returned in the response so the CI job shows
it. BOT_DEPLOY_AUTO=0 makes the server refuse. configure.py generates the secret, writes both settings into
/etc/vanguard/deploy.env, installs deployBotBundle.sh as /usr/local/bin/vanguard-deploy-bot and creates
/srv/bot/app; --show prints what to paste into Gitea.
Chose this over mounting the Docker socket into job containers, which would give every CI job root on the
host. Also moved threading/subprocess to dumpd's module imports, which the new module-level lock needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.
- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
image now ships none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bin/mtx-on-ready.sh (runOnReady inside the mediamtx container) posts the is-live notice and, for sources whose
audio is not Opus, runs ffmpeg: video copy, audio -> Opus, published over MediaMTX's loopback RTSP to
<user>/<Title>.webrtc (record: no). dumpd accepts .webrtc publishes only from 127.0.0.1. The portal hides
mirror paths, plays the mirror over WHEP (HLS stays on the source), holds an AAC source for 15 s while its
mirror comes up, and corrects the viewer count. Verified headless with a mocked path list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Moves existing panels (video keeps playing) into cols=ceil(sqrt(n)) row-major and equalizes group sizes.
Verified headless for 3-6 streams.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
CSS aspect-ratio on a flex:1 child was ignored, so object-fit:cover cropped arbitrarily. Player now
measures the panel (ResizeObserver: fires on every dockview resize/move), fits a frame of the game aspect,
lays the video across its width and hides the overflow at the bottom.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The published recording is the game picture + audio + tmcd track, nothing else. Streams without the
band are still remuxed losslessly. vgstrip plan exports TC_BAND and marks the sidecar band_removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
configure.py: three new host roles (files/sync/clips) in --full, --hosts, .env, Caddy generator;
main host serves about/ at the root (/about* redirects); /time only where it is fetched same-origin.
about/index.html links across hosts; addCommunityMediaServer.sh asks for the new names; Caddyfile.example
regenerated; README/FRESH-INSTALL/HOSTNAME-CHANGE (procedure C)/STATUS updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
