Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.
Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
with zero harvesters and no index. Container logs now use filestream + the container parser
with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.
Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
the user holds a privilege and that API does not exist when security is off (the
Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
the only gate. configure.py generates the service passwords, creates the accounts and writes
a root-only netrc for the deadman; --es-security retries just that step.
Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
that the stack's own alerting died.
Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
(loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.
Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for
Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers
host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and
Caddy's Prometheus endpoints.
Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box
whose spare capacity belongs to x264), single-node discovery, security off because it only listens on
loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start
without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart.
diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links.
The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is
not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when
space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually
written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the
disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty
webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord
notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still
evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in
/etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the
container's user while the repo copy cannot.
alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.
configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.
The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.
check.sh validates all five observability configs; confirmed it catches a deliberate break.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.
- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
image now ships none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
