Phase 13 build: logs, metrics, alerting and a deadman on a diag. host

Six containers, all loopback-published, reachable only through the gated diag. host:
  victorialogs      30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
  vector            docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
  victoriametrics   90d, scrapes the four taps that were already live plus the new services
  vmalert-metrics   11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
  vmalert-logs      5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
  alertmanager      groups and inhibits, posts to Discord, webhook read from a 0600 file not the config

The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.

Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 15:45:58 -06:00
parent 09882833a0
commit 8ae43905dc
13 changed files with 695 additions and 63 deletions

View File

@@ -12,6 +12,8 @@ POST /me/clips -> {"device_id": "..."}: create clips/<user> (+ .stignore), add
POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from
/etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1;
otherwise logs the push and answers 202 "auto-deploy disabled".
POST /heartbeat -> Alertmanager's Watchdog route: touches a file the deadman timer checks, so a dead
alerting pipeline becomes visible (bin/vanguard-deadman.sh).
POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle,
HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
@@ -326,6 +328,8 @@ def apply_bot_env(values):
log("bot_env_applied", keys=",".join(applied), refused=",".join(refused) or None)
return applied, refused
HEARTBEAT_FILE = Path(os.environ.get("ALERT_HEARTBEAT_FILE", "/srv/media/.markers/alert-heartbeat"))
_bot_deploy_lock = threading.Lock()
BOT_DEPLOY_MAX_BYTES = 300 * 1024 * 1024
BOT_DEPLOY_INSTALLER = os.environ.get("BOT_DEPLOY_INSTALLER", "/usr/local/bin/vanguard-deploy-bot")
@@ -493,6 +497,17 @@ class H(BaseHTTPRequestHandler):
"files": [str(f.relative_to(ROOT)) for f in files],
"gb": round(sum(f.stat().st_size for f in files) / 1e9, 2)})
def do_POST(self):
if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/heartbeat"):
# Alertmanager posts here every minute because the Watchdog alert always fires. Nothing is read from
# the body and nothing is authenticated: the port only accepts the Docker bridge, and the only thing
# this grants is keeping a timestamp fresh. The deadman treats a STALE file as the alarm.
try:
self.rfile.read(int(self.headers.get("Content-Length") or 0))
HEARTBEAT_FILE.parent.mkdir(parents=True, exist_ok=True)
HEARTBEAT_FILE.write_text(f"{int(_time.time())}\n")
except OSError as error:
return self._json(500, {"error": str(error)})
return self._json(200, {"ok": True})
if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/deploy-bot"):
# The bot's CI posts its release bundle here (bridge-only port; the firewall allows docker0/br-* alone).
# The body is the artifact; the signature proves it came from a runner holding BOT_DEPLOY_SECRET.