Elastic security, our own metrics, and an alert pipeline that can actually reach Discord
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
156
bin/dumpd.py
156
bin/dumpd.py
@@ -12,12 +12,17 @@ POST /me/clips -> {"device_id": "..."}: create clips/<user> (+ .stignore), add
|
||||
POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from
|
||||
/etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1;
|
||||
otherwise logs the push and answers 202 "auto-deploy disabled".
|
||||
POST /heartbeat -> Alertmanager's Watchdog route: touches a file the deadman timer checks, so a dead
|
||||
alerting pipeline becomes visible (bin/vanguard-deadman.sh).
|
||||
POST /heartbeat -> LEGACY, nothing calls this now. It was the Watchdog route when Alertmanager, and then a
|
||||
Kibana webhook connector, posted here. A basic Elastic licence has no webhook connector, so
|
||||
Kibana's rules write into the vanguard-alerts index instead and the deadman reads the
|
||||
heartbeat from there. Kept because it costs nothing and is a ready-made target if an
|
||||
external monitor is ever added.
|
||||
POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle,
|
||||
HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
|
||||
May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
|
||||
into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
|
||||
GET /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
|
||||
gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
|
||||
POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
|
||||
token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret.
|
||||
|
||||
@@ -420,6 +425,140 @@ def verify_token(token):
|
||||
return None
|
||||
return user.lower()
|
||||
|
||||
# ---- GET /metrics -----------------------------------------------------------------------------------
|
||||
# Prometheus text for Metricbeat's prometheus module. Two sources are merged:
|
||||
# 1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
|
||||
# 2. gauges computed here at scrape time: unpublished recordings, encodes in flight, failures, MediaMTX
|
||||
# Only the loopback listener serves it, and Caddy routes no host to /metrics (every dumpd route over there is
|
||||
# an explicit path matcher), so this is reachable from the host and from nowhere else. Nothing here is
|
||||
# authenticated and nothing here is allowed to fail: a scrape must never be able to disturb the stack.
|
||||
METRICS_DIR = Path(os.environ.get("VG_METRICS_DIR", str(MARK / "metrics")))
|
||||
MTX_API = os.environ.get("MTX_API_URL", "http://127.0.0.1:9997")
|
||||
IDLE_S = int(os.environ.get("MINS_IDLE", "3")) * 60 # the sweep's own "finished being written" rule
|
||||
|
||||
# counters key head -> (metric, type, help, the labels the remaining key fields fill, in order)
|
||||
COUNTER_MAP = {
|
||||
"jobs": ("vanguard_media_jobs_total", "counter", "normalise/deband jobs that finished", ("verdict", "outcome", "kind")),
|
||||
"seconds": ("vanguard_media_job_seconds_total", "counter", "wall-clock seconds spent in ffmpeg", ("verdict", "kind")),
|
||||
"runs": ("vanguard_media_runs_total", "counter", "times the script has started work", ()),
|
||||
"deband": ("vanguard_media_deband_total", "counter", "timecode bands cropped off", ()),
|
||||
"last_finish": ("vanguard_media_last_finish_timestamp_seconds", "gauge", "when the script last finished", ()),
|
||||
"last_duration": ("vanguard_media_last_run_seconds", "gauge", "how long the last finished run took", ()),
|
||||
}
|
||||
|
||||
def _num(value):
|
||||
"""Prometheus wants a plain number: %g would turn a unix timestamp into 1.76e+09."""
|
||||
return "%d" % value if float(value).is_integer() else "%.3f" % value
|
||||
|
||||
def _glob(*patterns):
|
||||
found = []
|
||||
for pattern in patterns:
|
||||
try:
|
||||
found.extend(ROOT.glob(pattern))
|
||||
except OSError:
|
||||
pass
|
||||
return found
|
||||
|
||||
def mtx_paths():
|
||||
"""MediaMTX's own view of what is live. None if it did not answer."""
|
||||
try:
|
||||
with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=1.5) as response:
|
||||
return json.load(response).get("items") or []
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def metrics_text():
|
||||
order, grouped = [], {}
|
||||
def add(metric, typ, helptext, labels, value):
|
||||
if metric not in grouped:
|
||||
grouped[metric] = (typ, helptext, [])
|
||||
order.append(metric)
|
||||
grouped[metric][2].append((labels, value))
|
||||
|
||||
# 1. whatever the scripts have counted
|
||||
for counters in sorted(METRICS_DIR.glob("*.counters")):
|
||||
component = counters.name[: -len(".counters")]
|
||||
try:
|
||||
lines = counters.read_text().splitlines()
|
||||
except OSError:
|
||||
continue
|
||||
for line in lines:
|
||||
field = line.split()
|
||||
if len(field) != 2:
|
||||
continue
|
||||
try:
|
||||
value = float(field[1])
|
||||
except ValueError:
|
||||
continue
|
||||
head, *rest = field[0].split(":")
|
||||
if head not in COUNTER_MAP:
|
||||
continue
|
||||
metric, typ, helptext, names = COUNTER_MAP[head]
|
||||
labels = {"component": component}
|
||||
labels.update(dict(zip(names, rest)))
|
||||
add(metric, typ, helptext, labels, value)
|
||||
|
||||
# 2. is each script executing right now? (its pid file exists only while it runs; a crash leaves a stale
|
||||
# one, so the pid is checked against /proc rather than trusted)
|
||||
for pid_file in sorted(METRICS_DIR.glob("*.pid")):
|
||||
try:
|
||||
pid = int(pid_file.read_text().strip() or 0)
|
||||
except (OSError, ValueError):
|
||||
pid = 0
|
||||
add("vanguard_media_running", "gauge", "1 while the script is executing",
|
||||
{"component": pid_file.name[: -len(".pid")]}, 1 if pid and Path(f"/proc/{pid}").is_dir() else 0)
|
||||
|
||||
# 3. the queue. Depth-limited globs on purpose: never a walk of the whole media tree.
|
||||
now = _time.time()
|
||||
recording, pending, oldest = 0, 0, 0.0
|
||||
for f in _glob("streams/*/*/.live/*"):
|
||||
if f.name.startswith(".") or not f.is_file():
|
||||
continue
|
||||
try:
|
||||
age = now - f.stat().st_mtime
|
||||
except OSError:
|
||||
continue
|
||||
# the sweep ignores anything touched in the last MINS_IDLE, so younger files are still being written
|
||||
if age <= IDLE_S:
|
||||
recording += 1
|
||||
else:
|
||||
pending += 1
|
||||
oldest = max(oldest, age)
|
||||
add("vanguard_media_recording", "gauge", "recordings MediaMTX is still writing into .live/", {}, recording)
|
||||
add("vanguard_media_pending", "gauge", "finished recordings waiting to be normalised", {"kind": "stream"}, pending)
|
||||
add("vanguard_media_pending_oldest_seconds", "gauge", "age of the oldest waiting recording", {}, round(oldest))
|
||||
add("vanguard_media_encoding_in_flight", "gauge", "ffmpeg output files open right now", {},
|
||||
len(_glob("streams/*/*.part", "streams/*/*/*.part", "streams/*/*/.live/*.part",
|
||||
"clips/*/*.part", "clips/*/*/*.part", "clips/*/*/*/*.part")))
|
||||
add("vanguard_media_failed_files", "gauge", "files parked with a normalize-failed log", {},
|
||||
len(_glob("streams/*/*.normalize-failed.log", "streams/*/*/*.normalize-failed.log",
|
||||
"clips/*/*.normalize-failed.log", "clips/*/*/*.normalize-failed.log",
|
||||
"clips/*/*/*/*.normalize-failed.log")))
|
||||
|
||||
# 4. MediaMTX. The Prometheus scrape of :9998 already has per-path detail; this is the count the portal
|
||||
# and the dock actually mean by "live", with the derived .webrtc audio mirrors separated out.
|
||||
paths = mtx_paths()
|
||||
add("vanguard_mediamtx_up", "gauge", "did the MediaMTX API answer this scrape", {}, 0 if paths is None else 1)
|
||||
if paths is not None:
|
||||
ready = [p for p in paths if p.get("ready")]
|
||||
add("vanguard_streams_active", "gauge", "streams publishing right now, mirrors excluded", {},
|
||||
sum(1 for p in ready if not str(p.get("name", "")).endswith(".webrtc")))
|
||||
add("vanguard_streams_mirrors", "gauge", "derived .webrtc audio mirrors running", {},
|
||||
sum(1 for p in ready if str(p.get("name", "")).endswith(".webrtc")))
|
||||
add("vanguard_stream_readers", "gauge", "viewers attached across all paths", {},
|
||||
sum(len(p.get("readers") or []) for p in ready))
|
||||
|
||||
out = []
|
||||
for metric in order:
|
||||
typ, helptext, series = grouped[metric]
|
||||
if helptext:
|
||||
out.append(f"# HELP {metric} {helptext}")
|
||||
out.append(f"# TYPE {metric} {typ}")
|
||||
for labels, value in series:
|
||||
tags = ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
|
||||
out.append(f"{metric}{{{tags}}} {_num(value)}" if tags else f"{metric} {_num(value)}")
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
server_version = "dumpd/1"
|
||||
def _json(self, code, obj):
|
||||
@@ -429,6 +568,13 @@ class H(BaseHTTPRequestHandler):
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
def _text(self, code, body):
|
||||
raw = body.encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(raw)))
|
||||
self.end_headers()
|
||||
self.wfile.write(raw)
|
||||
def _deny(self):
|
||||
self._json(401, {"error": "not authenticated (no identity header from the proxy)"})
|
||||
def _parse(self):
|
||||
@@ -457,6 +603,12 @@ class H(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.server.server_port == AUTH_LISTEN[1]:
|
||||
return self._json(404, {"error": "not found"}) # the auth port serves POST /mtx-auth only
|
||||
if self.path == "/metrics":
|
||||
try:
|
||||
return self._text(200, metrics_text())
|
||||
except Exception as error: # a scrape must never take dumpd down
|
||||
log("metrics_failed", level="error", error=str(error))
|
||||
return self._text(500, f"# metrics unavailable: {error}\n")
|
||||
if self.path.startswith("/time"):
|
||||
# server clock for the timecode strip (the /clock/ page and Go Live measure their offset against this)
|
||||
return self._json(200, {"now_ms": int(_time.time() * 1000)})
|
||||
|
||||
Reference in New Issue
Block a user