Rip out the Victoria stack for Elastic: four containers, Kibana as the single pane

Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for
Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers
host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and
Caddy's Prometheus endpoints.

Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box
whose spare capacity belongs to x264), single-node discovery, security off because it only listens on
loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start
without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart.

diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links.

The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is
not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when
space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually
written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the
disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 16:53:19 -06:00
parent 6f77764fec
commit 88bb28989c
13 changed files with 265 additions and 579 deletions

1
.gitignore vendored
View File

@@ -7,7 +7,6 @@ bot/app/__pycache__/
# generated by configure.py on the server — never commit or scp these
mediamtx.yml
copyparty.conf
observability/alertmanager.yml
Caddyfile
stream-token.secret
deploy.env