Rip out the Victoria stack for Elastic: four containers, Kibana as the single pane
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and Caddy's Prometheus endpoints. Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box whose spare capacity belongs to x264), single-node discovery, security off because it only listens on loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart. diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links. The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,16 +1,20 @@
|
||||
#!/bin/bash
|
||||
# The deadman: the one check that does not depend on the observability stack being alive.
|
||||
# The deadman: the one check that does not depend on the Elastic stack being alive.
|
||||
#
|
||||
# vmalert and Alertmanager tell you when something on the box is wrong. Nothing inside that chain can tell you
|
||||
# the chain itself is down, and silence looks exactly like health. So this runs on the HOST from a systemd
|
||||
# timer, uses nothing but curl, and posts straight to a Discord webhook.
|
||||
# Kibana tells you when something on the box is wrong. Nothing inside Elasticsearch, Kibana or the Beats can
|
||||
# tell you that they are the thing that broke, and silence looks exactly like health. So this runs on the HOST
|
||||
# from a systemd timer, uses nothing but curl, and posts straight to a Discord webhook.
|
||||
#
|
||||
# It checks three things:
|
||||
# 1. every piece of the stack answers its health endpoint
|
||||
# 2. Alertmanager's Watchdog heartbeat is fresh (proves rule evaluation AND routing work end to end,
|
||||
# not merely that the processes are up)
|
||||
# 3. the disks, because that is the failure this box is most likely to die of, and it must be reported
|
||||
# even when the thing that would normally report it is the thing that broke
|
||||
# It checks six things, in the order they go wrong in practice:
|
||||
# 1. Elasticsearch answers and the cluster is not red
|
||||
# 2. no index has been flipped to read-only by the disk watermark — the classic Elastic failure, because it
|
||||
# does NOT clear itself when space is freed, so ingestion stays silently dead until someone resets it
|
||||
# 3. Kibana reports itself available (it is the only UI; if it is down you are blind)
|
||||
# 4. Filebeat has actually written a document recently — "running" is not the same as "shipping"
|
||||
# 5. Metricbeat likewise
|
||||
# 6. the heartbeat a Kibana rule posts to dumpd is fresh, which proves rule evaluation AND delivery work
|
||||
# …and the disks, because that is the failure this box is most likely to die of, and it must be reported
|
||||
# even when the thing that would normally report it is what broke.
|
||||
#
|
||||
# State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes,
|
||||
# and says so once when everything recovers.
|
||||
@@ -20,38 +24,72 @@
|
||||
# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
|
||||
set -uo pipefail
|
||||
|
||||
ES="${ES_URL:-http://127.0.0.1:9200}"
|
||||
KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
|
||||
WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
|
||||
HEARTBEAT="${ALERT_HEARTBEAT_FILE:-/srv/media/.markers/alert-heartbeat}"
|
||||
HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-600}"
|
||||
HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
|
||||
STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
|
||||
STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
|
||||
DISK_PCT="${DEADMAN_DISK_PCT:-90}"
|
||||
|
||||
problems=()
|
||||
note() { problems+=("$1"); }
|
||||
jq_get() { python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: sys.exit(1)
|
||||
for k in sys.argv[1].split("."):
|
||||
d = d.get(k) if isinstance(d, dict) else None
|
||||
if d is None: sys.exit(1)
|
||||
print(d)' "$1" 2>/dev/null; }
|
||||
|
||||
alive() { # alive <label> <url>
|
||||
local code
|
||||
code=$(curl -s -o /dev/null -m 5 -w '%{http_code}' "$2" || true)
|
||||
[[ $code == 2* || $code == 3* ]] || note "$1 is not answering (HTTP ${code:-no response})"
|
||||
}
|
||||
|
||||
# Each service runs behind the path prefix it is published under on diag.HOST, so its health endpoint
|
||||
# lives under that prefix too.
|
||||
alive "VictoriaLogs" "http://127.0.0.1:9428/logs/health"
|
||||
alive "VictoriaMetrics" "http://127.0.0.1:8428/metrics-db/health"
|
||||
alive "vmalert (metrics)" "http://127.0.0.1:8880/rules-metrics/health"
|
||||
alive "vmalert (logs)" "http://127.0.0.1:8881/rules-logs/health"
|
||||
alive "Alertmanager" "http://127.0.0.1:9093/alerts/-/healthy"
|
||||
|
||||
# 2. the heartbeat. A missing file right after a restart is not yet a fault, so only age counts.
|
||||
if [[ -f $HEARTBEAT ]]; then
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
|
||||
(( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (vmalert → Alertmanager → dumpd path is broken)"
|
||||
# 1. Elasticsearch
|
||||
health=$(curl -s -m 8 "$ES/_cluster/health" || true)
|
||||
if [[ -z $health ]]; then
|
||||
note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
|
||||
else
|
||||
note "no alert heartbeat has ever arrived (is the Watchdog rule loaded and routed?)"
|
||||
status=$(printf '%s' "$health" | jq_get status)
|
||||
case "$status" in
|
||||
green|yellow) ;;
|
||||
red) note "Elasticsearch cluster health is RED: some index has no usable shard" ;;
|
||||
*) note "Elasticsearch health is unreadable (got: ${status:-nothing})" ;;
|
||||
esac
|
||||
|
||||
# 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
|
||||
blocked=$(curl -s -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
|
||||
| python3 -c 'import json,sys
|
||||
try: d=json.load(sys.stdin)
|
||||
except Exception: sys.exit(0)
|
||||
bad=[n for n,v in d.items() if str(v.get("settings",{}).get("index",{}).get("blocks",{}).get("read_only_allow_delete","")).lower()=="true"]
|
||||
print(len(bad))' 2>/dev/null || echo 0)
|
||||
(( ${blocked:-0} > 0 )) && note "$blocked Elasticsearch index/indices are READ-ONLY from the disk watermark; ingestion is dead until this is reset (see README 5h)"
|
||||
|
||||
# 4 and 5. are the Beats actually writing?
|
||||
for beat in filebeat metricbeat; do
|
||||
count=$(curl -s -m 8 -H 'Content-Type: application/json' \
|
||||
"$ES/${beat}-*/_count" \
|
||||
-d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
|
||||
if [[ -z ${count:-} ]]; then
|
||||
note "cannot tell whether $beat is shipping (no index yet, or the query failed)"
|
||||
elif (( count == 0 )); then
|
||||
note "$beat has written nothing in the last $((STALE_MAX_S/60)) minutes — it may be up but not shipping"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# 3. disks, independently of any exporter
|
||||
# 3. Kibana
|
||||
level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
|
||||
[[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
|
||||
|
||||
# 6. the heartbeat a Kibana rule posts here. Missing right after a restart is not yet a fault; age is.
|
||||
if [[ -f $HEARTBEAT ]]; then
|
||||
age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
|
||||
(( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (the Kibana rule → webhook → dumpd path is broken)"
|
||||
else
|
||||
note "no alert heartbeat has ever arrived (is the Kibana heartbeat rule created? README 5h)"
|
||||
fi
|
||||
|
||||
# the disks, independently of any agent
|
||||
while read -r pct mount; do
|
||||
(( pct >= DISK_PCT )) && note "$mount is ${pct}% full"
|
||||
done < <(df --output=pcent,target /srv / 2>/dev/null | tail -n +2 | tr -d '%' | awk '{print $1, $2}')
|
||||
@@ -61,10 +99,8 @@ now="$(IFS='|'; echo "${problems[*]-}")"
|
||||
mkdir -p "$(dirname "$STATE")"
|
||||
was="$(cat "$STATE" 2>/dev/null || true)"
|
||||
printf '%s' "$now" > "$STATE"
|
||||
[[ $now == "$was" ]] && exit 0 # nothing changed since the last run: stay quiet
|
||||
[[ $now == "$was" ]] && exit 0
|
||||
|
||||
# The webhook is optional. With none configured the checks still run and still land in the journal
|
||||
# (journalctl -u vanguard-deadman), so nothing silently stops working; only the Discord delivery is skipped.
|
||||
webhook=""
|
||||
if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
|
||||
case "$webhook" in
|
||||
@@ -73,7 +109,7 @@ case "$webhook" in
|
||||
*) echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
|
||||
esac
|
||||
|
||||
post() { # post <text>; a no-op without a webhook
|
||||
post() {
|
||||
[[ -n $webhook ]] || return 1
|
||||
curl -s -m 15 -H 'Content-Type: application/json' \
|
||||
--data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
|
||||
@@ -83,20 +119,15 @@ post() { # post <text>; a no-op witho
|
||||
}
|
||||
|
||||
if [[ -z $now ]]; then
|
||||
echo "deadman: recovered — the alerting stack and disks are healthy again"
|
||||
post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again." || true
|
||||
echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
|
||||
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf 'deadman: %s\n' "${problems[@]}" >&2
|
||||
body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside the stack):"
|
||||
body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside Elastic):"
|
||||
for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
|
||||
body+=$'\n'"Until this clears, assume no other alert can reach you."
|
||||
if post "$body"; then exit 0; fi
|
||||
|
||||
# Problems found and nobody could be told. Exit non-zero so the unit shows as failed: with no webhook
|
||||
# configured, `systemctl is-failed vanguard-deadman.service` is the only signal there is.
|
||||
if [[ -z $webhook ]]; then
|
||||
echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
|
||||
fi
|
||||
[[ -z $webhook ]] && echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user