techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Browse Source 
Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 19:58:06 -06:00
parent ef357c6d82
commit b38d40ab0c
20 changed files with 1322 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files

207
configure.py
View File

@@ -24,7 +24,8 @@ Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/va
Discord bot token/guild), folders, restarts
Set VG_DRY=1 to write files only (no sudo / docker / folders).
"""
import getpass, hashlib, os, re, secrets, shutil, subprocess, sys
import base64, getpass, hashlib, json, os, re, secrets, shutil, subprocess, sys, time
import urllib.error, urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
@@ -390,8 +391,23 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
env.setdefault("BOT_DEPLOY_AUTO", "1") # 0 = refuse bundles posted by CI; install by hand instead
env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID") # the only .env keys the bot's CI may set
env.setdefault("DIAG_REQUIRE_GROUP", "media-admins") # who may see logs, metrics and alerts
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook for Kibana rules AND the deadman
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook: vanguard-alerts.sh delivery AND the deadman
env.setdefault("KIBANA_ENCRYPTION_KEY", secrets.token_hex(32)) # without a fixed key Kibana loses saved objects on restart
# Elasticsearch security is ON (observability/kibana.yml explains why, and why nobody sees a login).
# These are service credentials, never typed by a person: Discord remains the only human login.
def pw(): return "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32]
env.setdefault("ES_ELASTIC_PASSWORD", pw()) # the elastic superuser; bootstrapped on first secured start
env.setdefault("ES_KIBANA_PASSWORD", pw()) # kibana_system: Kibana's own internal account
env.setdefault("VG_BEATS_USER", "vanguard_beats")
env.setdefault("VG_BEATS_PASSWORD", pw()) # what Filebeat and Metricbeat ship as
env.setdefault("VG_DIAG_USER", "vanguard_diag") # who every Kibana visitor silently becomes
env.setdefault("VG_DIAG_PASSWORD", pw())
# What that visitor may do. superuser keeps today's behaviour exactly (everyone past the gate is an admin).
# Narrowing this to "editor" is step one of the read-only tier in STATUS.md 9b.
env.setdefault("DIAG_KIBANA_ROLES", "superuser")
env.setdefault("ALERT_INDEX", "vanguard-alerts") # the alert bus: Kibana writes, vanguard-alerts.sh delivers
env.setdefault("LOG_RETENTION_DAYS", "30") # ILM delete phase for filebeat-*/metricbeat-*
env.setdefault("ALERT_RETENTION_DAYS", "90") # how long delivered alerts stay queryable
env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot
env.setdefault("DISCORD_GUILD_ID", "")
env["WATCH_HOST"] = watch_host or "watch.example.com"
@@ -421,7 +437,7 @@ rtspAddress: 127.0.0.1:8554
rtspTransports: [tcp]
rtspEncryption: "no"
moq: no
metrics: yes # Prometheus text format for whichever metrics store is chosen (LOGGING.md); loopback only
metrics: yes # Prometheus text format, scraped by Metricbeat (LOGGING.md); loopback only
metricsAddress: :9998
pprof: no
@@ -559,9 +575,10 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
e = read_env()
Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
# 0640 root:65534 — the alertmanager image runs as nobody(65534) and the deadman runs as root. A webhook
# URL is a credential for that channel, so it is not world-readable.
run(["sudo", "install", "-m", "0640", "-o", "root", "-g", "65534", "alert-webhook", "/etc/vanguard/alert-webhook"])
# 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be
# group-readable by nobody(65534) for the Alertmanager container, which no longer exists.
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh
if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
@@ -577,13 +594,27 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
Path("99-vanguard-elasticsearch.conf").write_text("vm.max_map_count=262144\n")
run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"]) # script counters -> dumpd /metrics
# The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
# /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
"bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
if media:
run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
if Path("bin/vgmetrics.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband
if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
# The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
run(["sudo", "systemctl", "restart", "dumpd"])
print("== containers")
dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"]
dc = compose_cmd()
# --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how
# the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them.
run(dc + ["up", "-d", "--build", "--remove-orphans"]) # recreates only services whose definition changed
@@ -594,10 +625,138 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
print(" run docker compose restart mediamtx between streams (cuts a live recording). Adding/removing")
print(" streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
def compose_cmd():
"""docker compose, with sudo only if this user cannot reach the daemon."""
if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:
return ["docker", "compose"]
return ["sudo", "docker", "compose"]
def es_security(dc):
"""Create Elasticsearch's native users after the cluster is up. Idempotent; safe to re-run.
Security is on, so three accounts have to exist before Kibana or the Beats can work:
kibana_system Kibana's internal account (built in; only its password needs setting)
VG_BEATS_USER what Filebeat and Metricbeat authenticate as
VG_DIAG_USER who every Kibana visitor silently becomes (kibana.yml's anonymous provider)
The elastic superuser itself is bootstrapped by the container from ELASTIC_PASSWORD, which only works
while the cluster has never had security on. If that did not take, this says exactly what to run rather
than guessing: a reset would desync the password the compose healthcheck uses.
"""
env = read_env()
elastic_pw = env.get("ES_ELASTIC_PASSWORD", "")
if not elastic_pw:
print(" elasticsearch: no ES_ELASTIC_PASSWORD in .env (write_env not run?) — skipped"); return
if DRY:
print(" (dry) elasticsearch: set kibana_system + create "
f"{env.get('VG_BEATS_USER')} and {env.get('VG_DIAG_USER')}"); return
base = "http://127.0.0.1:9200"
def call(method, path, body=None):
request = urllib.request.Request(base + path, method=method,
data=None if body is None else json.dumps(body).encode())
request.add_header("Content-Type", "application/json")
request.add_header("Authorization", "Basic " + base64.b64encode(f"elastic:{elastic_pw}".encode()).decode())
with urllib.request.urlopen(request, timeout=20) as response:
raw = response.read()
return json.loads(raw) if raw else {}
status = None
for _ in range(45): # the cluster can take a while on a box busy with x264
try:
call("GET", "/_cluster/health"); status = "ok"; break
except urllib.error.HTTPError as error:
status = error.code
if error.code == 401: break # answering, but our password is not the one it bootstrapped
time.sleep(4)
except Exception:
time.sleep(4)
if status == 401:
print(" elasticsearch: REFUSED the password in .env. The cluster already had security enabled, so")
print(" ELASTIC_PASSWORD no longer bootstraps anything. Set it to the real one:")
print(" docker compose exec elasticsearch bin/elasticsearch-reset-password -u elastic -b -s")
print(" put that value in .env as ES_ELASTIC_PASSWORD, then re-run --apply.")
return
if status != "ok":
print(" elasticsearch: not answering yet — re-run ./configure.py --apply once it is up"); return
try:
# If security is somehow still off, every call below 400s with "no handler found for uri". Check once
# and say so plainly, rather than reporting a confusing failure on the first password write.
call("POST", "/_security/user/_has_privileges", {"cluster": ["monitor"]})
except urllib.error.HTTPError as error:
if error.code == 400:
print(" elasticsearch: the security API is absent — this cluster is still running with")
print(" xpack.security.enabled=false. Recreate it: docker compose up -d elasticsearch")
return
try:
call("POST", "/_security/user/kibana_system/_password", {"password": env["ES_KIBANA_PASSWORD"]})
print(" elasticsearch: kibana_system password set")
for user, password, roles in (
(env["VG_BEATS_USER"], env["VG_BEATS_PASSWORD"], ["superuser"]),
(env["VG_DIAG_USER"], env["VG_DIAG_PASSWORD"],
[r.strip() for r in env.get("DIAG_KIBANA_ROLES", "superuser").split(",") if r.strip()]),
):
call("PUT", f"/_security/user/{user}", {"password": password, "roles": roles,
"full_name": "Vanguard service account"})
print(f" elasticsearch: user {user} -> {', '.join(roles)}")
except urllib.error.HTTPError as error:
print(f" elasticsearch: user setup failed ({error.code}) — {error.read()[:200].decode(errors='replace')}")
return
# The alert bus. Kibana's rules write here through the .index connector (the only outward connector a
# basic licence allows), bin/vanguard-alerts.sh delivers from here to Discord, and the deadman watches
# for the heartbeat document. Pre-created with an explicit mapping so "level" and "rule" are keywords
# and therefore aggregatable, rather than whatever dynamic mapping the first document implies.
alert_index = env.get("ALERT_INDEX", "vanguard-alerts")
try:
call("PUT", f"/{alert_index}", {
"settings": {"number_of_replicas": 0},
"mappings": {"properties": {
"@timestamp": {"type": "date"},
"kind": {"type": "keyword"},
"rule": {"type": "keyword"},
"level": {"type": "keyword"},
"service": {"type": "keyword"},
"message": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 1024}}},
}}})
print(f" elasticsearch: alert index {alert_index} created")
except urllib.error.HTTPError as error:
body = error.read().decode(errors="replace")
print(f" elasticsearch: alert index {alert_index} already there"
if "resource_already_exists" in body else f" elasticsearch: alert index failed — {body[:160]}")
# Retention. The Beats create their own ILM policies with a rollover but NO delete phase, so indices
# grow until the disk watermark turns them read-only — which the deadman reports but cannot undo. Inject
# a delete phase instead of writing a whole policy, so whatever the Beats chose for rollover is kept.
keep_days = env.get("LOG_RETENTION_DAYS", "30")
for policy in ("filebeat", "metricbeat"):
try:
current = call("GET", f"/_ilm/policy/{policy}")[policy]["policy"]
except (urllib.error.HTTPError, KeyError):
print(f" elasticsearch: ILM policy {policy} not created yet — re-run --apply after the beat has started")
continue
phases = current.setdefault("phases", {})
if phases.get("delete", {}).get("min_age") == f"{keep_days}d":
print(f" elasticsearch: {policy} already deletes after {keep_days}d")
continue
phases["delete"] = {"min_age": f"{keep_days}d", "actions": {"delete": {}}}
call("PUT", f"/_ilm/policy/{policy}", {"policy": current})
print(f" elasticsearch: {policy} indices now deleted after {keep_days}d")
# The deadman runs on the host as root and must authenticate too. netrc, not -u, so the password never
# shows up in anyone's process list.
Path("es-netrc").write_text(f"machine 127.0.0.1 login elastic password {elastic_pw}\n")
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "es-netrc", "/etc/vanguard/es-netrc"])
Path("es-netrc").unlink(missing_ok=True)
# Kibana and the Beats were started before those accounts existed, so they are sitting in a retry loop.
run(dc + ["restart", "kibana", "filebeat", "metricbeat"])
print(" elasticsearch: netrc written for the deadman; kibana + beats restarted")
def post_start_fixups(dc):
"""Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin."""
env = read_env()
print("== post-start fixups (idempotent)")
es_security(dc) # before anything else: Kibana and the Beats cannot authenticate until this runs
# Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it
cfg = Path("/srv/media/.syncthing/config.xml")
if not DRY and cfg.exists():
@@ -627,6 +786,32 @@ def post_start_fixups(dc):
print(f" gitea: admin user {env['GITEA_ADMIN_USER']} created (password in .env)")
else: print(" gitea: ok" if r.returncode == 0 else " gitea: not ready (install page or db?) — run ./configure.py --apply again later")
else: print(" (dry) gitea admin user")
# Kibana first-run state: the alert connector, the heartbeat rule and the data view. Without this a
# fresh install has no way to deliver an alert and the deadman alarms about its own missing heartbeat.
# It needs an API key because Basic is not an accepted scheme for this Kibana (anonymous is the only
# enabled provider), and it waits for Kibana itself, which es_security restarted a moment ago.
setup = Path("observability/kibana-setup.py")
if setup.exists() and env.get("ES_ELASTIC_PASSWORD"):
if DRY:
print(" (dry) kibana: connector, heartbeat rule, data view")
else:
try:
request = urllib.request.Request(
"http://127.0.0.1:9200/_security/api_key", method="POST",
data=json.dumps({"name": "vanguard-setup", "expiration": "10m"}).encode())
request.add_header("Content-Type", "application/json")
request.add_header("Authorization", "Basic " + base64.b64encode(
f"elastic:{env['ES_ELASTIC_PASSWORD']}".encode()).decode())
with urllib.request.urlopen(request, timeout=20) as response:
api_key = json.load(response)["encoded"]
result = subprocess.run([sys.executable, str(setup)], text=True, capture_output=True,
env={**os.environ, "KEY": api_key,
"ALERT_INDEX": env.get("ALERT_INDEX", "vanguard-alerts")})
for line in (result.stdout + result.stderr).strip().splitlines():
print(" kibana:", line)
except Exception as error:
print(f" kibana: first-run setup skipped ({error}) — run it by hand, see its docstring")
# Jellyfin plugin from the committed zip
if Path("installJellyfinPlugin.sh").exists():
if DRY: print(" (dry) installJellyfinPlugin.sh")
@@ -641,8 +826,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}")
print(f"== live : https://{live_host}/ (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live : (not set — run --full)")
alert_hook = read_env().get("ALERT_WEBHOOK_URL", "")
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); Kibana rules and the deadman post there" if alert_hook
else "NO WEBHOOK — Kibana has nowhere to send and the deadman only writes to the journal. Set ALERT_WEBHOOK_URL in .env."))
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); vanguard-alerts.timer and the deadman post there" if alert_hook
else "NO WEBHOOK — Kibana still records alerts in the vanguard-alerts index, but nothing delivers them: "
"vanguard-alerts.sh and the deadman write to the journal and exit non-zero. Set ALERT_WEBHOOK_URL in .env."))
print(f"== diag : https://{diag_host}/ (Kibana: logs, dashboards, alerting; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)")
print(f"== bot web : https://{bot_host}/ (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web : (not set — run --full)")
print(f"== gitea : https://{git_host}/ ssh port 2222 admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')} (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea : (not set — run --full)")
@@ -765,6 +951,11 @@ def main():
streamers, accounts = read_streamers(), read_accounts()
admin_pw = accounts.pop("admin", "")
editors = accounts
if mode == "--es-security":
# Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an
# --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but
# publishes nothing — three confusing symptoms with one cause.
es_security(compose_cmd()); return
if mode == "--show":
show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return
if mode == "--apply":