Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bin/vanguard-deadman.sh

@@ -25,9 +25,19 @@
 set -uo pipefail
 
 ES="${ES_URL:-http://127.0.0.1:9200}"
+# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
+# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
+# is missing, the calls below get 401 and the deadman reports Elasticsearch as unreachable — which is correct,
+# because from its point of view it is. Kibana needs nothing: its anonymous provider signs callers in.
+ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
+es_curl() {
+  if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi
+}
 KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
 WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
-HEARTBEAT="${ALERT_HEARTBEAT_FILE:-/srv/media/.markers/alert-heartbeat}"
+ALERT_INDEX="${ALERT_INDEX:-vanguard-alerts}"            # the alert bus Kibana's rules write into
+ALERT_STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"  # touched by every vanguard-alerts.sh run
+ALERT_STATE_MAX_S="${ALERT_STATE_MAX_S:-900}"
 HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
 STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
 STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
@@ -44,7 +54,7 @@ for k in sys.argv[1].split("."):
 print(d)' "$1" 2>/dev/null; }
 
 # 1. Elasticsearch
-health=$(curl -s -m 8 "$ES/_cluster/health" || true)
+health=$(es_curl -m 8 "$ES/_cluster/health" || true)
 if [[ -z $health ]]; then
   note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
 else
@@ -56,7 +66,7 @@ else
   esac
 
   # 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
-  blocked=$(curl -s -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
+  blocked=$(es_curl -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
             | python3 -c 'import json,sys
 try: d=json.load(sys.stdin)
 except Exception: sys.exit(0)
@@ -66,7 +76,7 @@ print(len(bad))' 2>/dev/null || echo 0)
 
   # 4 and 5. are the Beats actually writing?
   for beat in filebeat metricbeat; do
-    count=$(curl -s -m 8 -H 'Content-Type: application/json' \
+    count=$(es_curl -m 8 -H 'Content-Type: application/json' \
               "$ES/${beat}-*/_count" \
               -d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
     if [[ -z ${count:-} ]]; then
@@ -81,12 +91,31 @@ fi
 level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
 [[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
 
-# 6. the heartbeat a Kibana rule posts here. Missing right after a restart is not yet a fault; age is.
-if [[ -f $HEARTBEAT ]]; then
-  age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
-  (( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (the Kibana rule → webhook → dumpd path is broken)"
-else
-  note "no alert heartbeat has ever arrived (is the Kibana heartbeat rule created? README 5h)"
+# 6. the heartbeat. A Kibana rule writes a kind=heartbeat document into the alert index every minute (see
+# observability/kibana-setup.py). Its absence is the ONLY thing that catches the alerting pipeline itself
+# dying: if Kibana, its rule engine or the index connector stops, no ordinary alert can report it, because
+# the thing that would report it is what broke. Checked in Elasticsearch, not as a file: a basic licence has
+# no webhook connector, so nothing posts to dumpd any more.
+if [[ -n $health ]]; then
+  beats=$(es_curl -m 8 -H 'Content-Type: application/json' "$ES/$ALERT_INDEX/_count" \
+            -d "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"kind\":\"heartbeat\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-${HEARTBEAT_MAX_S}s\"}}}]}}}")
+  if grep -q 'index_not_found_exception' <<<"${beats:-}"; then
+    note "the alert index $ALERT_INDEX does not exist — the Kibana connector and heartbeat rule have not been created (observability/kibana-setup.py)"
+  else
+    hb=$(printf '%s' "${beats:-}" | jq_get count)
+    if [[ -z ${hb:-} ]]; then
+      note "cannot read the alert heartbeat (query against $ALERT_INDEX failed)"
+    elif (( hb == 0 )); then
+      note "no alert heartbeat in the last $((HEARTBEAT_MAX_S/60)) minutes — Kibana's rule engine or the index connector has stopped, so NO alert can reach Discord"
+    fi
+  fi
+fi
+
+# 7. is the delivery side running? The heartbeat proves Kibana still writes alerts; this proves something
+# still takes them to Discord. Both halves have to be alive for an alarm to arrive.
+if [[ -f $ALERT_STATE ]]; then
+  age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
+  (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
 fi
 
 # the disks, independently of any agent