Mentions
- bin/vgnotify.sh is shared by the alert timer and the deadman. A webhook will not ping a role
from <@&id> in the content alone: the request has to whitelist it, so every post now sends an
explicit allowed_mentions. parse is always [] as well, so an @everyone appearing inside a log
line or filename in an alert text can never ping the server.
- ALERT_MENTION is written as role:<id>/user:<id> rather than raw Discord syntax, and
ALERT_MENTION_LEVELS keeps pings to errors by default so warnings arrive quietly. The deadman
pings too, because its alarm means nothing else can get through, but its recovery is info level.
- Both scripts now read /etc/vanguard/alerts.env themselves instead of relying on systemd to
inject it. Running one by hand previously behaved differently from the timer, which is exactly
how "the mention is configured but never appears" happened.
configure.py --set
- Change one setting and apply only what it needs. --full re-prompts for every hostname and
secret, which is far too blunt for pointing the alarms at a new channel, and mistyping a
hostname there has broken TLS before.
- Every Discord credential and webhook is now one commented block at the top of .env, so there is
one place to look, and --show prints the same five together with the secrets truncated.
- Each key applies only its own side effect: the webhook rewrites its root-only file, the deploy
webhook rewrites deploy.env, the bot token recreates the bot container, log retention re-injects
the ILM delete phase. LIVE_WEBHOOK_URL deliberately stops short, because MediaMTX only reads it
at start and a recreate cuts a live recording.
Logout
- The proxy provider pointed at default-provider-invalidation-flow ("Logged out of application"),
which ends only that application's session and leaves the SSO session alive, so /logout dropped
the proxy cookie and forward-auth signed the user straight back in. It looked like a refresh.
default-invalidation-flow ("Logout") is what Authentik's own button uses. --apply now enforces
it had explicitly recommended the wrong flow.
Sweep
- The band-removal branch overwrote $reasocodec> marker, which is the
only signal telling the encode to convert audio. A Go Live stream with both the band and Opus
audio therefore got -c:a copy and shippeachable once Windows Go
Live audio started working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.
Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
with zero harvesters and no index. Container logs now use filestream + the container parser
with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.
Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
the user holds a privilege and that API does not exist when security is off (the
Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
the only gate. configure.py generates the service passwords, creates the accounts and writes
a root-only netrc for the deadman; --es-security retries just that step.
Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
that the stack's own alerting died.
Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
(loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.
Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
