Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
64 lines
2.7 KiB
Bash
64 lines
2.7 KiB
Bash
# Shared metric counters for the media scripts. Sourced, never executed.
|
|
#
|
|
# VG_MET_COMPONENT=sweep # a plain assignment, NOT a prefix on the next line:
|
|
# . /usr/local/bin/vgmetrics.sh && vgmet_init # bash discards "VAR=x . file" assignments on return
|
|
# vgmet_add "jobs:encode:done:stream" # +1
|
|
# vgmet_add "seconds:encode:stream" 41.2 # +41.2
|
|
#
|
|
# Counters accumulate across runs in MARK/metrics/<component>.counters as plain "key value" lines, and a
|
|
# <component>.pid file exists only while the script is executing. dumpd turns both into Prometheus text on
|
|
# its /metrics; nothing here renders or serves anything, so the scripts stay simple and there is one place
|
|
# that knows the metric names (bin/dumpd.py COUNTER_MAP).
|
|
#
|
|
# Keys are "name:label:label..." — the name picks the metric, the rest fill its labels in order.
|
|
# Writes are atomic (temp file + mv) and every failure is silent: metrics must never break a sweep.
|
|
|
|
VG_MET_DIR="${VG_MET_DIR:-${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
|
|
VG_MET_COMPONENT="${VG_MET_COMPONENT:-$(basename "${0%.sh}")}"
|
|
VG_MET_START=0
|
|
declare -A VGMET=()
|
|
|
|
vgmet_init() {
|
|
: "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
|
|
: "${VG_MET_COMPONENT:=unknown}"
|
|
VG_MET_START=$(date +%s)
|
|
mkdir -p "$VG_MET_DIR" 2>/dev/null || return 0
|
|
local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" k v
|
|
if [[ -r $f ]]; then
|
|
while read -r k v; do [[ -n ${k:-} && -n ${v:-} ]] && VGMET["$k"]="$v"; done < "$f"
|
|
fi
|
|
vgmet_add runs 1
|
|
printf '%s\n' "$$" > "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
|
|
trap vgmet_flush EXIT
|
|
}
|
|
|
|
# vgmet_add <key> [amount] — float-safe accumulate
|
|
vgmet_add() {
|
|
local k="$1" v="${2:-1}"
|
|
VGMET["$k"]=$(awk -v a="${VGMET[$k]:-0}" -v b="$v" 'BEGIN{ printf "%.3f", a+b }')
|
|
}
|
|
|
|
vgmet_set() { VGMET["$1"]="$2"; }
|
|
|
|
vgmet_flush() {
|
|
: "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
|
|
: "${VG_MET_COMPONENT:=unknown}"
|
|
local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" tmp now k
|
|
rm -f "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
|
|
[[ -d $VG_MET_DIR ]] || return 0
|
|
now=$(date +%s)
|
|
vgmet_set last_finish "$now"
|
|
vgmet_set last_duration "$(( now - VG_MET_START ))"
|
|
tmp=$(mktemp "$VG_MET_DIR/.$VG_MET_COMPONENT.XXXXXX" 2>/dev/null) || return 0
|
|
for k in "${!VGMET[@]}"; do printf '%s %s\n' "$k" "${VGMET[$k]}"; done > "$tmp" 2>/dev/null
|
|
chmod 0644 "$tmp" 2>/dev/null
|
|
mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" 2>/dev/null
|
|
}
|
|
|
|
# start/elapsed for one job
|
|
vgmet_time() { date +%s.%N; }
|
|
vgmet_since() { awk -v s="$1" -v e="$(date +%s.%N)" 'BEGIN{ printf "%.3f", e-s }'; }
|
|
|
|
# No-op fallbacks are defined by the callers if this file is missing, so a half-finished deploy cannot
|
|
# break normalising. Keep every function above failure-tolerant.
|