Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
54 lines
4.2 KiB
Bash
Executable File
54 lines
4.2 KiB
Bash
Executable File
#!/bin/bash
|
|
# Static checks for everything in the bundle. Run before committing / deploying.
|
|
set -uo pipefail
|
|
cd "$(dirname "$0")"
|
|
fail=0
|
|
say() { printf '%-28s %s\n' "$1" "$2"; }
|
|
|
|
for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh registerGiteaRunner.sh createGiteaDb.sh deployLive.sh addClipUser.sh deploy.sh deployBotBundle.sh switchToClone.sh firewall/firewall.sh bin/*.sh home/*.sh bot/entrypoint.sh mariadb/init/*.sh; do
|
|
[[ -f $f ]] || continue
|
|
bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; }
|
|
done
|
|
for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py observability/kibana-alarms.py; do
|
|
python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
|
|
done
|
|
rm -rf __pycache__ bin/__pycache__
|
|
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
|
|
node -e "
|
|
const fs=require('fs');
|
|
for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
|
|
new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" && say "dock + clock JS" ok || { say "dock + clock JS" "SYNTAX ERROR"; fail=1; }
|
|
diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null && say "vgclock.js in sync (dock/portal)" ok || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
|
|
# generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse
|
|
T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "
|
|
import configure as c, yaml
|
|
c.write_caddy('media.example.com','auth.example.com','watch.example.com','git.example.com','bot.example.com','live.example.com','files.example.com','sync.example.com','clips.example.com')
|
|
c.write_mediamtx('media.example.com'); yaml.safe_load(open('mediamtx.yml')); c.write_copyparty()
|
|
assert 'authMethod: http' in open('mediamtx.yml').read(); assert 'idp-h-usr' in open('copyparty.conf').read()
|
|
" ) && say "configure.py generators" ok || { say "configure.py generators" "FAILED"; fail=1; }
|
|
rm -rf "$T"
|
|
# dumpd GET /metrics: valid Prometheus text out of script counters + a fake media tree
|
|
T=$(mktemp -d); mkdir -p "$T/.markers/metrics" "$T/streams/u/Title/.live"
|
|
printf 'runs 2\njobs:encode:done:stream 1\nseconds:encode:stream 12.5\nlast_finish 1762210000\n' > "$T/.markers/metrics/sweep.counters"
|
|
touch -d '2020-01-01' "$T/streams/u/Title/.live/old.mp4"
|
|
if MEDIA_ROOT="$T" MTX_API_URL=http://127.0.0.1:1 python3 - <<'PYMET' >/dev/null 2>&1
|
|
import re, sys
|
|
sys.path.insert(0, "bin")
|
|
import dumpd
|
|
text = dumpd.metrics_text()
|
|
for line in text.splitlines():
|
|
assert line.startswith("#") or re.fullmatch(r'[a-z_]+(\{[^}]*\})? -?[0-9.]+', line), line
|
|
assert 'vanguard_media_jobs_total{component="sweep",kind="stream",outcome="done",verdict="encode"} 1' in text
|
|
assert 'vanguard_media_last_finish_timestamp_seconds{component="sweep"} 1762210000' in text # not 1.76e+09
|
|
assert 'vanguard_media_pending{kind="stream"} 1' in text
|
|
assert 'vanguard_mediamtx_up 0' in text and 'vanguard_streams_active' not in text # no fake zeros
|
|
PYMET
|
|
then say "dumpd /metrics" ok; else say "dumpd /metrics" "FAILED"; fail=1; fi
|
|
rm -rf "$T" bin/__pycache__
|
|
if command -v caddy >/dev/null; then
|
|
T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('media.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com','f.example.com','s.example.com','c.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; }
|
|
rm -rf "$T"
|
|
fi
|
|
[[ -d live/node_modules ]] && ( cd live && npx --no-install vite build --logLevel error >/dev/null 2>&1 ) && say "live portal build" ok || say "live portal build" "(skipped or failed — run npm run build in live/)"
|
|
exit $fail
|