Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
15
bin/dumpd.py
15
bin/dumpd.py
@@ -12,6 +12,8 @@ POST /me/clips -> {"device_id": "..."}: create clips/<user> (+ .stignore), add
|
||||
POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from
|
||||
/etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1;
|
||||
otherwise logs the push and answers 202 "auto-deploy disabled".
|
||||
POST /heartbeat -> Alertmanager's Watchdog route: touches a file the deadman timer checks, so a dead
|
||||
alerting pipeline becomes visible (bin/vanguard-deadman.sh).
|
||||
POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle,
|
||||
HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
|
||||
May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
|
||||
@@ -326,6 +328,8 @@ def apply_bot_env(values):
|
||||
log("bot_env_applied", keys=",".join(applied), refused=",".join(refused) or None)
|
||||
return applied, refused
|
||||
|
||||
HEARTBEAT_FILE = Path(os.environ.get("ALERT_HEARTBEAT_FILE", "/srv/media/.markers/alert-heartbeat"))
|
||||
|
||||
_bot_deploy_lock = threading.Lock()
|
||||
BOT_DEPLOY_MAX_BYTES = 300 * 1024 * 1024
|
||||
BOT_DEPLOY_INSTALLER = os.environ.get("BOT_DEPLOY_INSTALLER", "/usr/local/bin/vanguard-deploy-bot")
|
||||
@@ -493,6 +497,17 @@ class H(BaseHTTPRequestHandler):
|
||||
"files": [str(f.relative_to(ROOT)) for f in files],
|
||||
"gb": round(sum(f.stat().st_size for f in files) / 1e9, 2)})
|
||||
def do_POST(self):
|
||||
if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/heartbeat"):
|
||||
# Alertmanager posts here every minute because the Watchdog alert always fires. Nothing is read from
|
||||
# the body and nothing is authenticated: the port only accepts the Docker bridge, and the only thing
|
||||
# this grants is keeping a timestamp fresh. The deadman treats a STALE file as the alarm.
|
||||
try:
|
||||
self.rfile.read(int(self.headers.get("Content-Length") or 0))
|
||||
HEARTBEAT_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
HEARTBEAT_FILE.write_text(f"{int(_time.time())}\n")
|
||||
except OSError as error:
|
||||
return self._json(500, {"error": str(error)})
|
||||
return self._json(200, {"ok": True})
|
||||
if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/deploy-bot"):
|
||||
# The bot's CI posts its release bundle here (bridge-only port; the firewall allows docker0/br-* alone).
|
||||
# The body is the artifact; the signature proves it came from a runner holding BOT_DEPLOY_SECRET.
|
||||
|
||||
Reference in New Issue
Block a user