Rip out the Victoria stack for Elastic: four containers, Kibana as the single pane
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and Caddy's Prometheus endpoints. Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box whose spare capacity belongs to x264), single-node discovery, security off because it only listens on loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart. diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links. The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1
.gitignore
vendored
1
.gitignore
vendored
@@ -7,7 +7,6 @@ bot/app/__pycache__/
|
||||
# generated by configure.py on the server — never commit or scp these
|
||||
mediamtx.yml
|
||||
copyparty.conf
|
||||
observability/alertmanager.yml
|
||||
Caddyfile
|
||||
stream-token.secret
|
||||
deploy.env
|
||||
|
||||
@@ -1,16 +1,20 @@
|
||||
#!/bin/bash
|
||||
# The deadman: the one check that does not depend on the observability stack being alive.
|
||||
# The deadman: the one check that does not depend on the Elastic stack being alive.
|
||||
#
|
||||
# vmalert and Alertmanager tell you when something on the box is wrong. Nothing inside that chain can tell you
|
||||
# the chain itself is down, and silence looks exactly like health. So this runs on the HOST from a systemd
|
||||
# timer, uses nothing but curl, and posts straight to a Discord webhook.
|
||||
# Kibana tells you when something on the box is wrong. Nothing inside Elasticsearch, Kibana or the Beats can
|
||||
# tell you that they are the thing that broke, and silence looks exactly like health. So this runs on the HOST
|
||||
# from a systemd timer, uses nothing but curl, and posts straight to a Discord webhook.
|
||||
#
|
||||
# It checks three things:
|
||||
# 1. every piece of the stack answers its health endpoint
|
||||
# 2. Alertmanager's Watchdog heartbeat is fresh (proves rule evaluation AND routing work end to end,
|
||||
# not merely that the processes are up)
|
||||
# 3. the disks, because that is the failure this box is most likely to die of, and it must be reported
|
||||
# even when the thing that would normally report it is the thing that broke
|
||||
# It checks six things, in the order they go wrong in practice:
|
||||
# 1. Elasticsearch answers and the cluster is not red
|
||||
# 2. no index has been flipped to read-only by the disk watermark — the classic Elastic failure, because it
|
||||
# does NOT clear itself when space is freed, so ingestion stays silently dead until someone resets it
|
||||
# 3. Kibana reports itself available (it is the only UI; if it is down you are blind)
|
||||
# 4. Filebeat has actually written a document recently — "running" is not the same as "shipping"
|
||||
# 5. Metricbeat likewise
|
||||
# 6. the heartbeat a Kibana rule posts to dumpd is fresh, which proves rule evaluation AND delivery work
|
||||
# …and the disks, because that is the failure this box is most likely to die of, and it must be reported
|
||||
# even when the thing that would normally report it is what broke.
|
||||
#
|
||||
# State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes,
|
||||
# and says so once when everything recovers.
|
||||
@@ -20,38 +24,72 @@
|
||||
# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
|
||||
set -uo pipefail
|
||||
|
||||
ES="${ES_URL:-http://127.0.0.1:9200}"
|
||||
KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
|
||||
WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
|
||||
HEARTBEAT="${ALERT_HEARTBEAT_FILE:-/srv/media/.markers/alert-heartbeat}"
|
||||
HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-600}"
|
||||
HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
|
||||
STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
|
||||
STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
|
||||
DISK_PCT="${DEADMAN_DISK_PCT:-90}"
|
||||
|
||||
problems=()
|
||||
note() { problems+=("$1"); }
|
||||
jq_get() { python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: sys.exit(1)
|
||||
for k in sys.argv[1].split("."):
|
||||
d = d.get(k) if isinstance(d, dict) else None
|
||||
if d is None: sys.exit(1)
|
||||
print(d)' "$1" 2>/dev/null; }
|
||||
|
||||
alive() { # alive <label> <url>
|
||||
local code
|
||||
code=$(curl -s -o /dev/null -m 5 -w '%{http_code}' "$2" || true)
|
||||
[[ $code == 2* || $code == 3* ]] || note "$1 is not answering (HTTP ${code:-no response})"
|
||||
}
|
||||
|
||||
# Each service runs behind the path prefix it is published under on diag.HOST, so its health endpoint
|
||||
# lives under that prefix too.
|
||||
alive "VictoriaLogs" "http://127.0.0.1:9428/logs/health"
|
||||
alive "VictoriaMetrics" "http://127.0.0.1:8428/metrics-db/health"
|
||||
alive "vmalert (metrics)" "http://127.0.0.1:8880/rules-metrics/health"
|
||||
alive "vmalert (logs)" "http://127.0.0.1:8881/rules-logs/health"
|
||||
alive "Alertmanager" "http://127.0.0.1:9093/alerts/-/healthy"
|
||||
|
||||
# 2. the heartbeat. A missing file right after a restart is not yet a fault, so only age counts.
|
||||
if [[ -f $HEARTBEAT ]]; then
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
|
||||
(( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (vmalert → Alertmanager → dumpd path is broken)"
|
||||
# 1. Elasticsearch
|
||||
health=$(curl -s -m 8 "$ES/_cluster/health" || true)
|
||||
if [[ -z $health ]]; then
|
||||
note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
|
||||
else
|
||||
note "no alert heartbeat has ever arrived (is the Watchdog rule loaded and routed?)"
|
||||
status=$(printf '%s' "$health" | jq_get status)
|
||||
case "$status" in
|
||||
green|yellow) ;;
|
||||
red) note "Elasticsearch cluster health is RED: some index has no usable shard" ;;
|
||||
*) note "Elasticsearch health is unreadable (got: ${status:-nothing})" ;;
|
||||
esac
|
||||
|
||||
# 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
|
||||
blocked=$(curl -s -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
|
||||
| python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: sys.exit(0)
|
||||
bad=[n for n,v in d.items() if str(v.get("settings",{}).get("index",{}).get("blocks",{}).get("read_only_allow_delete","")).lower()=="true"]
|
||||
print(len(bad))' 2>/dev/null || echo 0)
|
||||
(( ${blocked:-0} > 0 )) && note "$blocked Elasticsearch index/indices are READ-ONLY from the disk watermark; ingestion is dead until this is reset (see README 5h)"
|
||||
|
||||
# 4 and 5. are the Beats actually writing?
|
||||
for beat in filebeat metricbeat; do
|
||||
count=$(curl -s -m 8 -H 'Content-Type: application/json' \
|
||||
"$ES/${beat}-*/_count" \
|
||||
-d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
|
||||
if [[ -z ${count:-} ]]; then
|
||||
note "cannot tell whether $beat is shipping (no index yet, or the query failed)"
|
||||
elif (( count == 0 )); then
|
||||
note "$beat has written nothing in the last $((STALE_MAX_S/60)) minutes — it may be up but not shipping"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# 3. disks, independently of any exporter
|
||||
# 3. Kibana
|
||||
level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
|
||||
[[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
|
||||
|
||||
# 6. the heartbeat a Kibana rule posts here. Missing right after a restart is not yet a fault; age is.
|
||||
if [[ -f $HEARTBEAT ]]; then
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
|
||||
(( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (the Kibana rule → webhook → dumpd path is broken)"
|
||||
else
|
||||
note "no alert heartbeat has ever arrived (is the Kibana heartbeat rule created? README 5h)"
|
||||
fi
|
||||
|
||||
# the disks, independently of any agent
|
||||
while read -r pct mount; do
|
||||
(( pct >= DISK_PCT )) && note "$mount is ${pct}% full"
|
||||
done < <(df --output=pcent,target /srv / 2>/dev/null | tail -n +2 | tr -d '%' | awk '{print $1, $2}')
|
||||
@@ -61,10 +99,8 @@ now="$(IFS='|'; echo "${problems[*]-}")"
|
||||
mkdir -p "$(dirname "$STATE")"
|
||||
was="$(cat "$STATE" 2>/dev/null || true)"
|
||||
printf '%s' "$now" > "$STATE"
|
||||
[[ $now == "$was" ]] && exit 0 # nothing changed since the last run: stay quiet
|
||||
[[ $now == "$was" ]] && exit 0
|
||||
|
||||
# The webhook is optional. With none configured the checks still run and still land in the journal
|
||||
# (journalctl -u vanguard-deadman), so nothing silently stops working; only the Discord delivery is skipped.
|
||||
webhook=""
|
||||
if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
|
||||
case "$webhook" in
|
||||
@@ -73,7 +109,7 @@ case "$webhook" in
|
||||
*) echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
|
||||
esac
|
||||
|
||||
post() { # post <text>; a no-op without a webhook
|
||||
post() {
|
||||
[[ -n $webhook ]] || return 1
|
||||
curl -s -m 15 -H 'Content-Type: application/json' \
|
||||
--data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
|
||||
@@ -83,20 +119,15 @@ post() { # post <text>; a no-op witho
|
||||
}
|
||||
|
||||
if [[ -z $now ]]; then
|
||||
echo "deadman: recovered — the alerting stack and disks are healthy again"
|
||||
post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again." || true
|
||||
echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
|
||||
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf 'deadman: %s\n' "${problems[@]}" >&2
|
||||
body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside the stack):"
|
||||
body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside Elastic):"
|
||||
for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
|
||||
body+=$'\n'"Until this clears, assume no other alert can reach you."
|
||||
if post "$body"; then exit 0; fi
|
||||
|
||||
# Problems found and nobody could be told. Exit non-zero so the unit shows as failed: with no webhook
|
||||
# configured, `systemctl is-failed vanguard-deadman.service` is the only signal there is.
|
||||
if [[ -z $webhook ]]; then
|
||||
echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
|
||||
fi
|
||||
[[ -z $webhook ]] && echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
|
||||
exit 1
|
||||
|
||||
2
check.sh
2
check.sh
@@ -13,7 +13,7 @@ for f in configure.py bin/dumpd.py bin/vgstrip.py; do
|
||||
python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
|
||||
done
|
||||
rm -rf __pycache__ bin/__pycache__
|
||||
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/vector.yaml observability/scrape.yml observability/alerts-metrics.yml observability/alerts-logs.yml observability/alertmanager.example.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
|
||||
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
|
||||
node -e "
|
||||
const fs=require('fs');
|
||||
for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
|
||||
|
||||
100
configure.py
100
configure.py
@@ -20,7 +20,7 @@ Reads : Caddyfile (hostnames), mediamtx.yml (streamers), .env
|
||||
(editors/admins are Authentik groups now: editors, media-admins, admins — nothing local)
|
||||
(these are GENERATED on the server and gitignored; *.example.* in the repo are reference only —
|
||||
never scp them over the live ones: that is how streamers got wiped once)
|
||||
Writes : Caddyfile, mediamtx.yml, copyparty.conf, observability/alertmanager.yml, stream-token.secret -> /etc/vanguard, .env (generated secrets +
|
||||
Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/vanguard, .env (generated secrets +
|
||||
Discord bot token/guild), folders, restarts
|
||||
Set VG_DRY=1 to write files only (no sudo / docker / folders).
|
||||
"""
|
||||
@@ -232,29 +232,10 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
|
||||
# vanguard:diag
|
||||
{diag_host} {{
|
||||
encode zstd gzip
|
||||
{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # VictoriaLogs runs with -http.pathPrefix=/logs and emits /logs/... URLs itself, so the prefix is
|
||||
# passed through unchanged. Stripping it would break the UI's own asset paths.
|
||||
redir /logs /logs/select/vmui/ 302
|
||||
redir /logs/ /logs/select/vmui/ 302
|
||||
handle /logs* {{
|
||||
reverse_proxy 127.0.0.1:9428
|
||||
}}
|
||||
handle /metrics-db* {{
|
||||
reverse_proxy 127.0.0.1:8428 # VictoriaMetrics UI + query API
|
||||
}}
|
||||
handle /alerts* {{
|
||||
reverse_proxy 127.0.0.1:9093 # Alertmanager: what is firing, silences
|
||||
}}
|
||||
handle /rules-metrics* {{
|
||||
reverse_proxy 127.0.0.1:8880 # vmalert over VictoriaMetrics
|
||||
}}
|
||||
handle /rules-logs* {{
|
||||
reverse_proxy 127.0.0.1:8881 # vmalert over VictoriaLogs
|
||||
}}
|
||||
handle {{
|
||||
root * /srv/diag
|
||||
try_files {{path}} /index.html
|
||||
file_server # diag/index.html: what each path above is
|
||||
{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there
|
||||
# is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published.
|
||||
reverse_proxy 127.0.0.1:5601 {{
|
||||
flush_interval -1
|
||||
}}
|
||||
}}
|
||||
"""
|
||||
@@ -409,7 +390,8 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
|
||||
env.setdefault("BOT_DEPLOY_AUTO", "1") # 0 = refuse bundles posted by CI; install by hand instead
|
||||
env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID") # the only .env keys the bot's CI may set
|
||||
env.setdefault("DIAG_REQUIRE_GROUP", "media-admins") # who may see logs, metrics and alerts
|
||||
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook for Alertmanager AND the deadman
|
||||
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook for Kibana rules AND the deadman
|
||||
env.setdefault("KIBANA_ENCRYPTION_KEY", secrets.token_hex(32)) # without a fixed key Kibana loses saved objects on restart
|
||||
env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot
|
||||
env.setdefault("DISCORD_GUILD_ID", "")
|
||||
env["WATCH_HOST"] = watch_host or "watch.example.com"
|
||||
@@ -497,59 +479,6 @@ paths:
|
||||
~^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$: {{}}
|
||||
""")
|
||||
|
||||
def write_alertmanager():
|
||||
"""Alertmanager config. Generated, because the Discord receiver must be ABSENT when no webhook is set:
|
||||
a receiver with no notifiers is valid and simply drops notifications, whereas pointing discord_configs at
|
||||
an empty webhook_url_file may stop Alertmanager from loading at all. The URL itself stays out of this file
|
||||
and in /etc/vanguard/alert-webhook (0640 root:65534), which is what webhook_url_file reads."""
|
||||
webhook = read_env().get("ALERT_WEBHOOK_URL", "").strip()
|
||||
discord = """ discord_configs:
|
||||
- webhook_url_file: /etc/vanguard/alert-webhook
|
||||
title: '{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}'
|
||||
message: |
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ if .Annotations.detail }}{{ .Annotations.detail }}
|
||||
{{ end }}{{ end }}
|
||||
""" if webhook.startswith("https://") else """ # No ALERT_WEBHOOK_URL in .env, so this receiver sends nowhere: alerts still group and still show at
|
||||
# https://DIAG_HOST/alerts, they just are not delivered. Set the variable and re-apply to turn it on.
|
||||
"""
|
||||
Path("observability/alertmanager.yml").write_text(f"""# Alertmanager — GENERATED by configure.py from ALERT_WEBHOOK_URL. Reference copy: alertmanager.example.yml
|
||||
# The heartbeat route is half of the deadman: the Watchdog alert always fires, so Alertmanager posts to dumpd
|
||||
# every minute and bin/vanguard-deadman.sh treats a STALE heartbeat as the alarm.
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: discord
|
||||
group_by: [alertname, service]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
routes:
|
||||
- matchers: [severity="none"]
|
||||
receiver: heartbeat
|
||||
group_wait: 0s
|
||||
group_interval: 1m
|
||||
repeat_interval: 1m
|
||||
- matchers: [severity="critical"]
|
||||
receiver: discord
|
||||
group_wait: 10s
|
||||
repeat_interval: 1h
|
||||
|
||||
inhibit_rules:
|
||||
- source_matchers: [alertname="MediaDiskCritical"]
|
||||
target_matchers: [alertname="MediaDiskFilling"]
|
||||
- source_matchers: [alertname="ContainerGone"]
|
||||
target_matchers: [alertname="ContainerRestartLoop"]
|
||||
|
||||
receivers:
|
||||
- name: discord
|
||||
{discord} - name: heartbeat
|
||||
webhook_configs:
|
||||
- url: http://host.docker.internal:8081/heartbeat
|
||||
send_resolved: false
|
||||
""")
|
||||
|
||||
def write_copyparty():
|
||||
"""copyparty behind Caddy + Authentik: identity and groups arrive as headers; permissions are per group."""
|
||||
key = read_env().get("COPYPARTY_IDP_KEY", "key")
|
||||
@@ -610,7 +539,7 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
|
||||
if not yes(" continue anyway?", False): print("aborted; add streamers first ([s] in the menu)"); sys.exit(1)
|
||||
print("\n== writing configs")
|
||||
write_env(watch_host, discord_token, discord_guild, git_host, bot_host, live_host, domain, live_webhook, files_host, sync_host, clips_host, diag_host)
|
||||
write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); write_mediamtx(domain); write_copyparty(); write_alertmanager()
|
||||
write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); write_mediamtx(domain); write_copyparty()
|
||||
Path("stream-token.secret").write_text(read_env()["STREAM_TOKEN_SECRET"] + "\n"); os.chmod("stream-token.secret", 0o600)
|
||||
run(["sudo", "install", "-d", "-o", os.environ.get("SUDO_USER") or os.environ.get("USER") or "debian", "/srv/live"])
|
||||
print("== app data folders")
|
||||
@@ -642,8 +571,11 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
|
||||
print(" (deploy pipeline files missing — deploy.sh / systemd/deploy-pending.* not uploaded; skipping vanguard-deploy install)")
|
||||
run(["sudo", "install", "-m", "0755", "deployBotBundle.sh", "/usr/local/bin/vanguard-deploy-bot"]) # dumpd runs this for POST /deploy-bot
|
||||
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/bot", "/srv/bot/app"]) # the bot's released bundle -> /app
|
||||
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/victorialogs", "/srv/appdata/victoriametrics"])
|
||||
run(["sudo", "install", "-d", "-o", "65534", "-g", "65534", "/srv/appdata/alertmanager"]) # prom/alertmanager runs as nobody
|
||||
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/elasticsearch"]) # the ES image runs as uid 1000
|
||||
# Elasticsearch refuses to start below this; it is a host setting, not a container one.
|
||||
run(["sudo", "sysctl", "-q", "-w", "vm.max_map_count=262144"])
|
||||
Path("99-vanguard-elasticsearch.conf").write_text("vm.max_map_count=262144\n")
|
||||
run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
|
||||
run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state
|
||||
if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
|
||||
run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
|
||||
@@ -707,9 +639,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
|
||||
if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}")
|
||||
print(f"== live : https://{live_host}/ (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live : (not set — run --full)")
|
||||
alert_hook = read_env().get("ALERT_WEBHOOK_URL", "")
|
||||
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); deadman posts there too" if alert_hook
|
||||
else "NO WEBHOOK — alerts and the deadman only write to the journal. Set ALERT_WEBHOOK_URL in .env."))
|
||||
print(f"== diag : https://{diag_host}/logs (logs, metrics, alerts; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)")
|
||||
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); Kibana rules and the deadman post there" if alert_hook
|
||||
else "NO WEBHOOK — Kibana has nowhere to send and the deadman only writes to the journal. Set ALERT_WEBHOOK_URL in .env."))
|
||||
print(f"== diag : https://{diag_host}/ (Kibana: logs, dashboards, alerting; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)")
|
||||
print(f"== bot web : https://{bot_host}/ (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web : (not set — run --full)")
|
||||
print(f"== gitea : https://{git_host}/ ssh port 2222 admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')} (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea : (not set — run --full)")
|
||||
print(f"== authentik: https://{auth_host}/ akadmin password = AUTHENTIK_BOOTSTRAP_PASSWORD in .env (set on first start; change it in the UI)" if auth_host else "== authentik: (not set — run --full)")
|
||||
|
||||
@@ -314,165 +314,99 @@ services:
|
||||
- gitea
|
||||
restart: unless-stopped
|
||||
|
||||
# ---- metrics taps (LOGGING.md): Prometheus text format on loopback, scraped by whichever store is chosen later
|
||||
node_exporter:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: node_exporter, vanguard.category: infra }
|
||||
image: quay.io/prometheus/node-exporter:latest
|
||||
container_name: node_exporter
|
||||
network_mode: host # listens on 127.0.0.1:9100 only
|
||||
pid: host
|
||||
command: ["--path.rootfs=/host", "--web.listen-address=127.0.0.1:9100", "--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run|var/lib/docker/.+)($$|/)"]
|
||||
volumes:
|
||||
- /:/host:ro,rslave
|
||||
restart: unless-stopped
|
||||
|
||||
cadvisor:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: cadvisor, vanguard.category: infra }
|
||||
image: gcr.io/cadvisor/cadvisor:latest
|
||||
container_name: cadvisor
|
||||
ports:
|
||||
- "127.0.0.1:9101:8080" # per-container CPU / memory / network / disk
|
||||
command: ["--housekeeping_interval=10s", "--docker_only=true", "--store_container_labels=false", "--whitelisted_container_labels=vanguard.service,vanguard.category"]
|
||||
volumes:
|
||||
- /:/rootfs:ro
|
||||
- /var/run:/var/run:ro
|
||||
- /sys:/sys:ro
|
||||
- /var/lib/docker/:/var/lib/docker:ro
|
||||
- /dev/disk/:/dev/disk:ro
|
||||
devices:
|
||||
- /dev/kmsg
|
||||
privileged: true
|
||||
restart: unless-stopped
|
||||
|
||||
# ================================================================ observability (Phase 13)
|
||||
# Logs: Vector -> VictoriaLogs. Metrics: VictoriaMetrics scrapes the four taps.
|
||||
# Alerting: a vmalert per datasource -> Alertmanager -> Discord. All loopback; published via diag.HOST.
|
||||
# The deadman (bin/vanguard-deadman.sh, a host timer) watches THIS stack, because nothing in it can.
|
||||
victorialogs:
|
||||
# ================================================================ observability — Elastic (Phase 13)
|
||||
# One store, one UI, one collector pair. Replaces VictoriaLogs+Vector+VictoriaMetrics+2×vmalert+Alertmanager
|
||||
# +node_exporter+cAdvisor: eight containers for four, and Kibana is the single pane for logs, dashboards and
|
||||
# alerting. Everything binds loopback and is published only through diag.HOST behind the Authentik gate.
|
||||
elasticsearch:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: victorialogs, vanguard.category: infra }
|
||||
image: docker.io/victoriametrics/victoria-logs:latest
|
||||
container_name: victorialogs
|
||||
user: "1000:1000"
|
||||
command:
|
||||
- -storageDataPath=/data
|
||||
- -retentionPeriod=30d
|
||||
- -httpListenAddr=:9428
|
||||
- -http.pathPrefix=/logs # its own UI then emits /logs/... so Caddy must NOT strip the prefix
|
||||
ports:
|
||||
- "127.0.0.1:9428:9428"
|
||||
volumes:
|
||||
- /srv/appdata/victorialogs:/data
|
||||
restart: unless-stopped
|
||||
|
||||
vector:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: vector, vanguard.category: infra }
|
||||
image: docker.io/timberio/vector:latest-debian
|
||||
container_name: vector
|
||||
labels: { vanguard.service: elasticsearch, vanguard.category: infra }
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:9.2.0
|
||||
container_name: elasticsearch
|
||||
environment:
|
||||
VECTOR_CONFIG: /etc/vector/vector.yaml
|
||||
# Pinned deliberately. A JVM that sizes its own heap on a 62 GB box would take a quarter of it, on a
|
||||
# machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics.
|
||||
ES_JAVA_OPTS: -Xms2g -Xmx2g
|
||||
discovery.type: single-node
|
||||
xpack.security.enabled: "false" # loopback only, behind Caddy + Authentik; no second login
|
||||
xpack.ml.enabled: "false" # not used, and it reserves memory
|
||||
cluster.routing.allocation.disk.watermark.low: 85%
|
||||
cluster.routing.allocation.disk.watermark.high: 90%
|
||||
cluster.routing.allocation.disk.watermark.flood_stage: 95%
|
||||
bootstrap.memory_lock: "true"
|
||||
ulimits:
|
||||
memlock: { soft: -1, hard: -1 }
|
||||
nofile: { soft: 65536, hard: 65536 }
|
||||
ports:
|
||||
- "127.0.0.1:9200:9200"
|
||||
volumes:
|
||||
- ./observability/vector.yaml:/etc/vector/vector.yaml:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock # Phase 13 decision "A": container logs need the Docker
|
||||
# API, so this is root-equivalent on the host. Upgrade
|
||||
# paths: journald log driver, or a read-only socket proxy.
|
||||
- /srv/appdata/elasticsearch:/usr/share/elasticsearch/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
|
||||
interval: 20s
|
||||
timeout: 10s
|
||||
retries: 12
|
||||
start_period: 90s
|
||||
restart: unless-stopped
|
||||
|
||||
kibana:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: kibana, vanguard.category: infra }
|
||||
image: docker.elastic.co/kibana/kibana:9.2.0
|
||||
container_name: kibana
|
||||
environment:
|
||||
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
|
||||
SERVER_PUBLICBASEURL: https://${DIAG_HOST:-diag.example.com}
|
||||
SERVER_BASEPATH: "" # served at the root of diag.HOST, so no prefix rewriting to get wrong
|
||||
TELEMETRY_OPTIN: "false"
|
||||
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
XPACK_REPORTING_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
ports:
|
||||
- "127.0.0.1:5601:5601"
|
||||
depends_on:
|
||||
elasticsearch: { condition: service_healthy }
|
||||
restart: unless-stopped
|
||||
|
||||
filebeat:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: filebeat, vanguard.category: infra }
|
||||
image: docker.elastic.co/beats/filebeat:9.2.0
|
||||
container_name: filebeat
|
||||
user: root
|
||||
command: ["filebeat", "-e", "--strict.perms=false"]
|
||||
volumes:
|
||||
- ./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock # autodiscover reads container names and our vanguard.* labels
|
||||
- /var/log/journal:/var/log/journal:ro # our own systemd units
|
||||
- /etc/machine-id:/etc/machine-id:ro # journald needs it to find the right journal
|
||||
- vector_data:/var/lib/vector
|
||||
- /etc/machine-id:/etc/machine-id:ro
|
||||
- filebeat_data:/usr/share/filebeat/data
|
||||
depends_on:
|
||||
- victorialogs
|
||||
elasticsearch: { condition: service_healthy }
|
||||
restart: unless-stopped
|
||||
|
||||
victoriametrics:
|
||||
metricbeat:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: victoriametrics, vanguard.category: infra }
|
||||
image: docker.io/victoriametrics/victoria-metrics:latest
|
||||
container_name: victoriametrics
|
||||
user: "1000:1000"
|
||||
command:
|
||||
- -storageDataPath=/data
|
||||
- -retentionPeriod=90d
|
||||
- -httpListenAddr=:8428
|
||||
- -http.pathPrefix=/metrics-db # served at diag.HOST/metrics-db, so its own UI must emit that prefix
|
||||
- -promscrape.config=/etc/vm/scrape.yml
|
||||
- -promscrape.config.strictParse=false
|
||||
ports:
|
||||
- "127.0.0.1:8428:8428"
|
||||
labels: { vanguard.service: metricbeat, vanguard.category: infra }
|
||||
image: docker.elastic.co/beats/metricbeat:9.2.0
|
||||
container_name: metricbeat
|
||||
user: root
|
||||
command: ["metricbeat", "-e", "--strict.perms=false"]
|
||||
# system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards.
|
||||
network_mode: host # real interfaces, and loopback reach to MediaMTX/Caddy/ES/Kibana
|
||||
pid: host
|
||||
volumes:
|
||||
- /srv/appdata/victoriametrics:/data
|
||||
- ./observability/scrape.yml:/etc/vm/scrape.yml:ro
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway" # the four taps bind the host's loopback
|
||||
restart: unless-stopped
|
||||
|
||||
vmalert-metrics:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: vmalert-metrics, vanguard.category: infra }
|
||||
image: docker.io/victoriametrics/vmalert:latest
|
||||
container_name: vmalert-metrics
|
||||
command:
|
||||
- -datasource.url=http://victoriametrics:8428/metrics-db
|
||||
- -notifier.url=http://alertmanager:9093/alerts
|
||||
- -remoteWrite.url=http://victoriametrics:8428/metrics-db
|
||||
- -rule=/etc/alerts/alerts-metrics.yml
|
||||
- -httpListenAddr=:8880
|
||||
- -http.pathPrefix=/rules-metrics
|
||||
- -evaluationInterval=1m
|
||||
ports:
|
||||
- "127.0.0.1:8880:8880"
|
||||
volumes:
|
||||
- ./observability/alerts-metrics.yml:/etc/alerts/alerts-metrics.yml:ro
|
||||
- ./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- /proc:/hostfs/proc:ro
|
||||
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
|
||||
- /:/hostfs:ro
|
||||
- metricbeat_data:/usr/share/metricbeat/data
|
||||
depends_on:
|
||||
- victoriametrics
|
||||
- alertmanager
|
||||
restart: unless-stopped
|
||||
|
||||
vmalert-logs:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: vmalert-logs, vanguard.category: infra }
|
||||
image: docker.io/victoriametrics/vmalert:latest
|
||||
container_name: vmalert-logs
|
||||
# A vmalert instance has one datasource, so log rules need their own: this one queries VictoriaLogs'
|
||||
# LogsQL stats endpoints, which answer in the Prometheus format vmalert expects.
|
||||
command:
|
||||
- -datasource.url=http://victorialogs:9428/logs
|
||||
- -notifier.url=http://alertmanager:9093/alerts
|
||||
- -remoteWrite.url=http://victoriametrics:8428/metrics-db
|
||||
- -rule=/etc/alerts/alerts-logs.yml
|
||||
- -httpListenAddr=:8881
|
||||
- -http.pathPrefix=/rules-logs
|
||||
- -evaluationInterval=2m
|
||||
ports:
|
||||
- "127.0.0.1:8881:8881"
|
||||
volumes:
|
||||
- ./observability/alerts-logs.yml:/etc/alerts/alerts-logs.yml:ro
|
||||
depends_on:
|
||||
- victorialogs
|
||||
- alertmanager
|
||||
restart: unless-stopped
|
||||
|
||||
alertmanager:
|
||||
logging: *vglog
|
||||
labels: { vanguard.service: alertmanager, vanguard.category: infra }
|
||||
image: docker.io/prom/alertmanager:latest
|
||||
container_name: alertmanager
|
||||
command:
|
||||
- --config.file=/etc/alertmanager/alertmanager.yml
|
||||
- --storage.path=/data
|
||||
- --web.listen-address=:9093
|
||||
- --web.route-prefix=/alerts
|
||||
- --web.external-url=https://${DIAG_HOST:-diag.example.com}/alerts
|
||||
ports:
|
||||
- "127.0.0.1:9093:9093"
|
||||
volumes:
|
||||
- ./observability/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
|
||||
- /etc/vanguard/alert-webhook:/etc/vanguard/alert-webhook:ro # written by configure.py, mode 0600
|
||||
- /srv/appdata/alertmanager:/data
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway" # the Watchdog heartbeat goes to dumpd on the bridge port
|
||||
elasticsearch: { condition: service_healthy }
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
@@ -480,6 +414,7 @@ networks:
|
||||
|
||||
volumes:
|
||||
copyparty_hist:
|
||||
vector_data:
|
||||
filebeat_data:
|
||||
metricbeat_data:
|
||||
caddy_data:
|
||||
caddy_config:
|
||||
|
||||
@@ -42,9 +42,11 @@ install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/aut
|
||||
install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
|
||||
install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
|
||||
install -d -o 1000 -g 1000 /srv/bot /srv/bot/app # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
|
||||
install -d -o 1000 -g 1000 /srv/appdata/victorialogs /srv/appdata/victoriametrics # run as uid 1000
|
||||
install -d -o 65534 -g 65534 /srv/appdata/alertmanager # prom/alertmanager runs as nobody
|
||||
install -d -o 1000 -g 1000 /srv/appdata/elasticsearch # the ES image runs as uid 1000
|
||||
install -d /var/lib/vanguard # deadman state
|
||||
# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
|
||||
sysctl -q -w vm.max_map_count=262144
|
||||
printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
|
||||
install -d -o "$OWNER" /srv/live # live portal static build (deployLive.sh)
|
||||
install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
|
||||
for d in /srv/media/clips/*/; do
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
# REFERENCE COPY ONLY — configure.py generates observability/alertmanager.yml from ALERT_WEBHOOK_URL and that
|
||||
# generated file is what runs (it omits the Discord receiver's notifier entirely when no webhook is set, so
|
||||
# the config stays valid and alerts simply are not delivered). Never scp this over the generated one.
|
||||
#
|
||||
# Alertmanager: groups what vmalert fires and posts it to Discord. The webhook URL is NOT in here — it is a
|
||||
# secret, so configure.py writes it to /etc/vanguard/alert-webhook with mode 0600 and this reads it from there.
|
||||
#
|
||||
# The "heartbeat" route is the other half of the deadman. The Watchdog alert in alerts-metrics.yml always fires,
|
||||
# so Alertmanager posts a heartbeat to a file-watching receiver every interval; bin/vanguard-deadman.sh on the
|
||||
# host notices when that stops and tells Discord itself. Nothing inside the stack can report its own death.
|
||||
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: discord
|
||||
group_by: [alertname, service]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
routes:
|
||||
- matchers: [severity="none"]
|
||||
receiver: heartbeat
|
||||
group_wait: 0s
|
||||
group_interval: 1m
|
||||
repeat_interval: 1m
|
||||
- matchers: [severity="critical"]
|
||||
receiver: discord
|
||||
group_wait: 10s
|
||||
repeat_interval: 1h
|
||||
|
||||
inhibit_rules:
|
||||
# While the disk is critical, do not also shout that it is merely filling.
|
||||
- source_matchers: [alertname="MediaDiskCritical"]
|
||||
target_matchers: [alertname="MediaDiskFilling"]
|
||||
# If a core container is gone, its restart-loop alert is noise.
|
||||
- source_matchers: [alertname="ContainerGone"]
|
||||
target_matchers: [alertname="ContainerRestartLoop"]
|
||||
|
||||
receivers:
|
||||
- name: discord
|
||||
discord_configs:
|
||||
- webhook_url_file: /etc/vanguard/alert-webhook
|
||||
title: '{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}'
|
||||
message: |
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ if .Annotations.detail }}{{ .Annotations.detail }}
|
||||
{{ end }}{{ end }}
|
||||
- name: heartbeat
|
||||
# Goes to dumpd on the host (bridge-only port), which touches a file. bin/vanguard-deadman.sh checks that
|
||||
# file is fresh, so a heartbeat never reaches Discord but its ABSENCE does.
|
||||
webhook_configs:
|
||||
- url: http://127.0.0.1:9095/heartbeat
|
||||
send_resolved: false
|
||||
@@ -1,33 +0,0 @@
|
||||
# Rules vmalert evaluates against VictoriaLogs' /select/logsql/stats_query endpoints (LogsQL, not PromQL).
|
||||
groups:
|
||||
- name: logs
|
||||
interval: 2m
|
||||
type: vlogs
|
||||
rules:
|
||||
- alert: SweepFailing
|
||||
# normalize-sweep writes event=failed and leaves a .normalize-failed.log beside the file.
|
||||
expr: 'component:=sweep level:=error event:=failed | stats count() as hits'
|
||||
for: 1m
|
||||
labels: {severity: warning}
|
||||
annotations:
|
||||
summary: "the normalize sweep failed {{ $value }} time(s) in the last 2m"
|
||||
detail: "A recording or clip did not convert. Look for *.normalize-failed.log under /srv/media."
|
||||
- alert: PublishDenied
|
||||
# A few are normal (expired token, wrong prefix). A burst means a streamer is locked out.
|
||||
expr: 'component:=dumpd event:=publish_deny | stats count() as hits'
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "{{ $value }} publish denials in 2m — a streamer may be unable to go live"}
|
||||
- alert: BotDeployFailed
|
||||
expr: 'component:=dumpd event:=bot_deploy_exit level:=error | stats count() as hits'
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "a Discord bot release failed to install"}
|
||||
- alert: ServiceErrors
|
||||
expr: 'level:=error | stats by (service) count() as hits'
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "{{ $labels.service }} logged {{ $value }} errors in 2m"}
|
||||
- alert: CertificateTrouble
|
||||
expr: 'service:=caddy _msg:"obtain" _msg:"error" | stats count() as hits'
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "Caddy could not obtain or renew a certificate"}
|
||||
@@ -1,84 +0,0 @@
|
||||
# Rules vmalert evaluates against VictoriaMetrics. Metric conditions only; log conditions are in alerts-logs.yml.
|
||||
groups:
|
||||
- name: disk
|
||||
interval: 1m
|
||||
rules:
|
||||
- alert: MediaDiskFilling
|
||||
# The most likely quiet death on this box: recordings fill /srv and the sweep starts failing.
|
||||
expr: 100 - (node_filesystem_avail_bytes{mountpoint="/srv"} / node_filesystem_size_bytes{mountpoint="/srv"} * 100) > 85
|
||||
for: 10m
|
||||
labels: {severity: warning}
|
||||
annotations:
|
||||
summary: "/srv is {{ $value | printf \"%.0f\" }}% full"
|
||||
detail: "Recordings and clips live here. Check /srv/media/streams and the 15-day archive window."
|
||||
- alert: MediaDiskCritical
|
||||
expr: 100 - (node_filesystem_avail_bytes{mountpoint="/srv"} / node_filesystem_size_bytes{mountpoint="/srv"} * 100) > 94
|
||||
for: 2m
|
||||
labels: {severity: critical}
|
||||
annotations:
|
||||
summary: "/srv is {{ $value | printf \"%.0f\" }}% full — recording will fail"
|
||||
- alert: RootDiskFilling
|
||||
expr: 100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) > 85
|
||||
for: 10m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "/ is {{ $value | printf \"%.0f\" }}% full (docker images, journal, appdata)"}
|
||||
|
||||
- name: host
|
||||
interval: 1m
|
||||
rules:
|
||||
- alert: LoadSustained
|
||||
# The sweep is niced, so a sustained load above core count means something is not yielding.
|
||||
expr: node_load15 > count(count by (cpu) (node_cpu_seconds_total))
|
||||
for: 20m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "load15 {{ $value | printf \"%.1f\" }} above core count for 20m"}
|
||||
- alert: MemoryLow
|
||||
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
|
||||
for: 10m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "only {{ $value | printf \"%.0f\" }}% memory available"}
|
||||
- alert: RaidDegraded
|
||||
expr: node_md_disks{state!="active"} > 0
|
||||
for: 5m
|
||||
labels: {severity: critical}
|
||||
annotations: {summary: "md RAID has a disk that is not active"}
|
||||
|
||||
- name: containers
|
||||
interval: 1m
|
||||
rules:
|
||||
- alert: ContainerRestartLoop
|
||||
expr: increase(container_start_time_seconds{name!=""}[15m]) > 0 and changes(container_start_time_seconds{name!=""}[1h]) > 2
|
||||
for: 5m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "{{ $labels.name }} restarted more than twice in an hour"}
|
||||
- alert: ContainerGone
|
||||
# cAdvisor stops reporting a container that is not running at all.
|
||||
expr: absent(container_last_seen{name="mediamtx"}) or absent(container_last_seen{name="caddy"}) or absent(container_last_seen{name="discord-bot"})
|
||||
for: 5m
|
||||
labels: {severity: critical}
|
||||
annotations: {summary: "a core container is not running (mediamtx, caddy or discord-bot)"}
|
||||
|
||||
- name: streaming
|
||||
interval: 1m
|
||||
rules:
|
||||
- alert: StreamDroppingPackets
|
||||
expr: increase(mediamtx_paths_bytes_received{}[5m]) == 0 and mediamtx_paths{state="ready"} > 0
|
||||
for: 10m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "a MediaMTX path is ready but has received nothing for 10m"}
|
||||
|
||||
- name: observability
|
||||
interval: 1m
|
||||
rules:
|
||||
- alert: LogIngestionStalled
|
||||
# If this fires, the log half is blind — and if it cannot fire, the deadman covers it.
|
||||
expr: increase(vl_rows_ingested_total[15m]) == 0
|
||||
for: 15m
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: "VictoriaLogs has ingested nothing for 15m — is Vector running?"}
|
||||
- alert: Watchdog
|
||||
# Always firing on purpose: Alertmanager routes it to the heartbeat receiver, and the deadman script
|
||||
# on the host notices when it stops arriving. This is how the monitoring stack reports its own death.
|
||||
expr: vector(1)
|
||||
labels: {severity: none}
|
||||
annotations: {summary: "alerting pipeline is alive"}
|
||||
50
observability/filebeat.yml
Normal file
50
observability/filebeat.yml
Normal file
@@ -0,0 +1,50 @@
|
||||
# Filebeat: container logs + our own systemd units -> Elasticsearch.
|
||||
# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
|
||||
# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
|
||||
# needs no parsing. See LOGGING.md for what each service's lines look like.
|
||||
|
||||
filebeat.autodiscover:
|
||||
providers:
|
||||
- type: docker
|
||||
hints.enabled: false
|
||||
templates:
|
||||
- condition: ~ # every container
|
||||
config:
|
||||
- type: container
|
||||
paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
|
||||
|
||||
filebeat.inputs:
|
||||
- type: journald
|
||||
id: vanguard-units
|
||||
include_matches:
|
||||
match:
|
||||
- "_SYSTEMD_UNIT=dumpd.service"
|
||||
- "_SYSTEMD_UNIT=normalize-sweep.service"
|
||||
- "_SYSTEMD_UNIT=trash-purge.service"
|
||||
- "_SYSTEMD_UNIT=vgdeband.service"
|
||||
- "_SYSTEMD_UNIT=vanguard-deadman.service"
|
||||
|
||||
processors:
|
||||
- add_docker_metadata: ~
|
||||
- add_host_metadata: ~
|
||||
# promote our labels to real fields so Kibana filters on service.name and vanguard.category
|
||||
- copy_fields:
|
||||
fields:
|
||||
- from: container.labels.vanguard_service
|
||||
to: service.name
|
||||
- from: container.labels.vanguard_category
|
||||
to: vanguard.category
|
||||
fail_on_error: false
|
||||
ignore_missing: true
|
||||
- drop_fields:
|
||||
fields: ["agent.ephemeral_id", "ecs.version", "input.type", "log.offset"]
|
||||
ignore_missing: true
|
||||
|
||||
output.elasticsearch:
|
||||
hosts: ["http://elasticsearch:9200"]
|
||||
|
||||
setup.kibana:
|
||||
host: "http://kibana:5601"
|
||||
setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h)
|
||||
|
||||
logging.level: info
|
||||
40
observability/metricbeat.yml
Normal file
40
observability/metricbeat.yml
Normal file
@@ -0,0 +1,40 @@
|
||||
# Metricbeat: host and per-container metrics, replacing node_exporter and cAdvisor, plus the Prometheus
|
||||
# endpoints the stack already exposes. Runs on the host network so the system module sees the real interfaces
|
||||
# and so it can reach MediaMTX, Caddy, Elasticsearch and Kibana on loopback.
|
||||
|
||||
metricbeat.modules:
|
||||
- module: system
|
||||
period: 30s
|
||||
metricsets: [cpu, load, memory, network, process, process_summary, socket_summary, filesystem, fsstat, diskio, uptime]
|
||||
processes: ['.*']
|
||||
process.include_top_n:
|
||||
by_cpu: 10
|
||||
by_memory: 10
|
||||
system.hostfs: /hostfs # the host's real filesystems, not the container's
|
||||
|
||||
- module: docker
|
||||
period: 30s
|
||||
hosts: ["unix:///var/run/docker.sock"]
|
||||
metricsets: [container, cpu, diskio, healthcheck, info, memory, network]
|
||||
|
||||
- module: prometheus # MediaMTX: per path bytes/s, readers, publishers, dropped packets
|
||||
period: 30s
|
||||
metrics_path: /metrics
|
||||
hosts: ["http://127.0.0.1:9998"]
|
||||
|
||||
- module: prometheus # Caddy: requests by host and status, durations, TLS
|
||||
period: 60s
|
||||
metrics_path: /metrics
|
||||
hosts: ["http://127.0.0.1:2019"]
|
||||
|
||||
processors:
|
||||
- add_host_metadata: ~
|
||||
|
||||
output.elasticsearch:
|
||||
hosts: ["http://127.0.0.1:9200"]
|
||||
|
||||
setup.kibana:
|
||||
host: "http://127.0.0.1:5601"
|
||||
setup.dashboards.enabled: true # the system and docker dashboards, so there is something to look at
|
||||
|
||||
logging.level: info
|
||||
@@ -1,30 +0,0 @@
|
||||
# What VictoriaMetrics scrapes. Every target is loopback-only on this host (LOGGING.md "Metrics taps").
|
||||
global:
|
||||
scrape_interval: 30s
|
||||
external_labels:
|
||||
host: vanguard
|
||||
|
||||
scrape_configs:
|
||||
- job_name: host # CPU, RAM, disk space + I/O, network, load, md RAID
|
||||
static_configs: [{targets: ["host.docker.internal:9100"]}]
|
||||
- job_name: containers # per container CPU/memory/network/block IO, labelled vanguard.*
|
||||
static_configs: [{targets: ["host.docker.internal:9101"]}]
|
||||
- job_name: mediamtx # per path bytes/s, readers, publishers, dropped packets
|
||||
static_configs: [{targets: ["host.docker.internal:9998"]}]
|
||||
- job_name: caddy # requests by host and status, durations, TLS
|
||||
static_configs: [{targets: ["host.docker.internal:2019"]}]
|
||||
- job_name: victorialogs
|
||||
static_configs: [{targets: ["victorialogs:9428"]}]
|
||||
metrics_path: /logs/metrics # it runs behind -http.pathPrefix=/logs
|
||||
- job_name: victoriametrics
|
||||
static_configs: [{targets: ["127.0.0.1:8428"]}]
|
||||
metrics_path: /metrics-db/metrics
|
||||
- job_name: alertmanager
|
||||
static_configs: [{targets: ["alertmanager:9093"]}]
|
||||
metrics_path: /alerts/metrics
|
||||
- job_name: vmalert
|
||||
static_configs: [{targets: ["vmalert-metrics:8880"], labels: {rules: metrics}}]
|
||||
metrics_path: /rules-metrics/metrics
|
||||
- job_name: vmalert-logs
|
||||
static_configs: [{targets: ["vmalert-logs:8881"], labels: {rules: logs}}]
|
||||
metrics_path: /rules-logs/metrics
|
||||
@@ -1,102 +0,0 @@
|
||||
# Vector: collects from the two taps in LOGGING.md and ships to VictoriaLogs.
|
||||
#
|
||||
# docker_logs every container's stdout/stderr, with the vanguard.service / vanguard.category labels
|
||||
# we put on each one, so filtering by category needs no parsing
|
||||
# journald our own systemd units (dumpd, the sweep, trash purge, deploy pending, vgdeband)
|
||||
#
|
||||
# TRADE-OFF, chosen deliberately (CommunityDiscordMedia.md Phase 13): docker_logs needs the Docker API, so this
|
||||
# container holds the socket and therefore root-equivalent access to the host — the same exposure act_runner
|
||||
# already has. Accepted "for now". The upgrade paths, in order of preference, are the journald log driver for
|
||||
# every container (one tap, no socket, labels survive as journal fields) or a read-only socket proxy in front
|
||||
# of this container. Neither changes anything downstream of here.
|
||||
#
|
||||
# Every event ends up with: service, category, source_kind, message, level where it could be determined.
|
||||
|
||||
data_dir: /var/lib/vector
|
||||
|
||||
sources:
|
||||
containers:
|
||||
type: docker_logs
|
||||
docker_host: unix:///var/run/docker.sock
|
||||
exclude_containers: [vector] # never ship our own output: it would feed itself
|
||||
units:
|
||||
type: journald
|
||||
# systemd 250-257 reject reading past boots through journalctl, so this stays true: only the current boot
|
||||
# is shipped. Older entries are still on the host for `journalctl --boot=-1`.
|
||||
current_boot_only: true
|
||||
journal_directory: /var/log/journal
|
||||
include_units: [dumpd, normalize-sweep, trash-purge, deploy-pending, vgdeband]
|
||||
|
||||
transforms:
|
||||
# ---- one shape, whichever tap it came from
|
||||
normalised:
|
||||
type: remap
|
||||
inputs: [containers, units]
|
||||
source: |
|
||||
if exists(.container_name) {
|
||||
.service = .label."vanguard.service" || .container_name
|
||||
.category = .label."vanguard.category" || "infra"
|
||||
.source_kind = "container"
|
||||
del(.label); del(.container_id); del(.image); del(.container_created_at); del(.stream)
|
||||
} else {
|
||||
unit = ._SYSTEMD_UNIT || .unit || "host"
|
||||
.service = replace(string!(unit), ".service", "")
|
||||
.category = if .service == "dumpd" { "stream" } else { "media" }
|
||||
.source_kind = "unit"
|
||||
if !exists(.message) && exists(.MESSAGE) { .message = del(.MESSAGE) }
|
||||
# the journal carries a lot of bookkeeping; keep what a query or a human needs
|
||||
del(.PRIORITY); del(.SYSLOG_IDENTIFIER); del(.SYSLOG_FACILITY); del(._TRANSPORT)
|
||||
del(._PID); del(._UID); del(._GID); del(._COMM); del(._EXE); del(._CMDLINE)
|
||||
del(._CAP_EFFECTIVE); del(._SELINUX_CONTEXT); del(._AUDIT_SESSION); del(._AUDIT_LOGINUID)
|
||||
del(._SYSTEMD_CGROUP); del(._SYSTEMD_SLICE); del(._SYSTEMD_INVOCATION_ID)
|
||||
del(._BOOT_ID); del(._MACHINE_ID); del(._SOURCE_REALTIME_TIMESTAMP); del(._SYSTEMD_UNIT)
|
||||
}
|
||||
|
||||
# ---- parse the shapes LOGGING.md documents; anything else stays a plain message
|
||||
parsed:
|
||||
type: remap
|
||||
inputs: [normalised]
|
||||
source: |
|
||||
text = string(.message) ?? ""
|
||||
if starts_with(text, "{") {
|
||||
# caddy and authentik emit JSON
|
||||
fields, err = parse_json(text)
|
||||
if err == null {
|
||||
.json = fields
|
||||
.level = get(fields, ["level"]) ?? get(fields, ["severity"]) ?? null
|
||||
.message = get(fields, ["msg"]) ?? get(fields, ["event"]) ?? text
|
||||
}
|
||||
} else if match(text, r'(^|\s)(ts|level|component|event)=') {
|
||||
# our own scripts, and syncthing's key=value tail
|
||||
fields, err = parse_logfmt(text)
|
||||
if err == null {
|
||||
.kv = fields
|
||||
.level = get(fields, ["level"]) ?? null
|
||||
.event = get(fields, ["event"]) ?? null
|
||||
}
|
||||
} else if match(text, r'^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} ') {
|
||||
# mediamtx: 2026/11/03 05:00:00 INF [component] message
|
||||
fields, err = parse_regex(text, r'^\S+ \S+ (?P<lvl>\w+) (?P<rest>.*)$')
|
||||
if err == null { .level = fields.lvl; .message = fields.rest }
|
||||
} else if match(text, r'^\[\d{2}:\d{2}:\d{2}') {
|
||||
# jellyfin: [00:00:00.000] [LVL] [thread] Logger: message (stack traces arrive as their own events)
|
||||
fields, err = parse_regex(text, r'^\[[\d:.]+\] \[(?P<lvl>\w+)\] \[\d+\] (?P<rest>.*)$')
|
||||
if err == null { .level = fields.lvl; .message = fields.rest }
|
||||
}
|
||||
if exists(.level) { .level = downcase(string!(.level)) }
|
||||
|
||||
sinks:
|
||||
victorialogs:
|
||||
type: elasticsearch
|
||||
inputs: [parsed]
|
||||
# -http.pathPrefix=/logs applies to EVERY endpoint, ingestion included, so the bulk path lives under it
|
||||
endpoints: ["http://victorialogs:9428/logs/insert/elasticsearch/"]
|
||||
mode: bulk
|
||||
api_version: v8
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
query:
|
||||
_msg_field: message
|
||||
_time_field: timestamp
|
||||
_stream_fields: service,category,source_kind
|
||||
Reference in New Issue
Block a user