Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for
Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers
host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and
Caddy's Prometheus endpoints.
Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box
whose spare capacity belongs to x264), single-node discovery, security off because it only listens on
loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start
without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart.
diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links.
The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is
not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when
space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually
written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the
disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
VictoriaLogs' -http.pathPrefix applies to every endpoint, not just the UI, so the elasticsearch bulk path is
/logs/insert/elasticsearch/. It rejected the writes with exactly that message. The other three references
(vmalert datasource, the metrics scrape, the deadman health probe) were already prefixed correctly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
systemd 250-257 refuse to read past boots through journalctl, which made the source config invalid and put
the container in a restart loop with exit 78. The object!() casts were only warnings: inside the err == null
branch VRL has already narrowed the parsed value, so get() takes it directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each service now runs behind the prefix it is published under (-http.pathPrefix, or route-prefix plus
external-url for Alertmanager). Without that, three of the four UIs would have loaded at their sub-path and
then fetched their assets from the domain root, so the pages would have come up broken. The prefixes cascade,
so the scrape paths, vmalert's notifier and remote-write URLs, and the deadman's health probes all follow.
The index is diag/index.html mounted into Caddy, not an inline respond: Caddy reads {...} as placeholders and
the page's CSS is full of braces, so inline would have been mangled. It also matches how the gateway, dock and
clips pages are already served.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty
webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord
notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still
evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in
/etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the
container's user while the repo copy cannot.
alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six containers, all loopback-published, reachable only through the gated diag. host:
victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
victoriametrics 90d, scrapes the four taps that were already live plus the new services
vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config
The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.
Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
