Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.
Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
with zero harvesters and no index. Container logs now use filestream + the container parser
with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.
Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
the user holds a privilege and that API does not exist when security is off (the
Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
the only gate. configure.py generates the service passwords, creates the accounts and writes
a root-only netrc for the deadman; --es-security retries just that step.
Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
that the stack's own alerting died.
Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
(loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.
Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six containers, all loopback-published, reachable only through the gated diag. host:
victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
victoriametrics 90d, scrapes the four taps that were already live plus the new services
vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config
The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.
Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
