Alarm mentions, targeted --set, and a logout that actually logs out
Mentions
- bin/vgnotify.sh is shared by the alert timer and the deadman. A webhook will not ping a role
from <@&id> in the content alone: the request has to whitelist it, so every post now sends an
explicit allowed_mentions. parse is always [] as well, so an @everyone appearing inside a log
line or filename in an alert text can never ping the server.
- ALERT_MENTION is written as role:<id>/user:<id> rather than raw Discord syntax, and
ALERT_MENTION_LEVELS keeps pings to errors by default so warnings arrive quietly. The deadman
pings too, because its alarm means nothing else can get through, but its recovery is info level.
- Both scripts now read /etc/vanguard/alerts.env themselves instead of relying on systemd to
inject it. Running one by hand previously behaved differently from the timer, which is exactly
how "the mention is configured but never appears" happened.
configure.py --set
- Change one setting and apply only what it needs. --full re-prompts for every hostname and
secret, which is far too blunt for pointing the alarms at a new channel, and mistyping a
hostname there has broken TLS before.
- Every Discord credential and webhook is now one commented block at the top of .env, so there is
one place to look, and --show prints the same five together with the secrets truncated.
- Each key applies only its own side effect: the webhook rewrites its root-only file, the deploy
webhook rewrites deploy.env, the bot token recreates the bot container, log retention re-injects
the ILM delete phase. LIVE_WEBHOOK_URL deliberately stops short, because MediaMTX only reads it
at start and a recreate cuts a live recording.
Logout
- The proxy provider pointed at default-provider-invalidation-flow ("Logged out of application"),
which ends only that application's session and leaves the SSO session alive, so /logout dropped
the proxy cookie and forward-auth signed the user straight back in. It looked like a refresh.
default-invalidation-flow ("Logout") is what Authentik's own button uses. --apply now enforces
it had explicitly recommended the wrong flow.
Sweep
- The band-removal branch overwrote $reasocodec> marker, which is the
only signal telling the encode to convert audio. A Go Live stream with both the band and Opus
audio therefore got -c:a copy and shippeachable once Windows Go
Live audio started working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -63,7 +63,14 @@ probe() {
|
||||
afr=$(jq -r '[.streams[]|select(.codec_type=="video")][0].avg_frame_rate // "0/1"' <<<"$json")
|
||||
acodecs=$(jq -r '[.streams[]|select(.codec_type=="audio")|.codec_name]|unique|join(",")' <<<"$json")
|
||||
[[ $vcodec == none ]] && { echo "skip no-video"; return; }
|
||||
local fps; fps=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$afr")
|
||||
# Take the LOWER of r_frame_rate and avg_frame_rate. Either is unreliable alone: avg is garbage on a
|
||||
# fragmented or looped recording (one reported 95 fps for a 30 fps source, which sent it down the fps-cap
|
||||
# path and broke the encode), and r is a huge nominal value on genuine VFR. The lower of the two is the
|
||||
# honest answer to "is this really above 60".
|
||||
local fps; fps=$(awk -v rfr="$rfr" -v afr="$afr" 'BEGIN{
|
||||
split(rfr, a, "/"); split(afr, b, "/")
|
||||
r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
|
||||
if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
|
||||
local reasons=()
|
||||
[[ $vcodec == h264 ]] || reasons+=("codec=$vcodec")
|
||||
[[ $pix == yuv420p ]] || reasons+=("pix=$pix")
|
||||
@@ -113,6 +120,11 @@ while IFS= read -r -d '' src; do
|
||||
[[ -e "$src" ]] || continue # dumped meanwhile
|
||||
|
||||
read -r verdict reason <<<"$(probe "$src")"
|
||||
# Keep what the PROBE said: the deband branch below overwrites $reason with its own text, which silently
|
||||
# loses the "audio=<codec>" marker. That marker is the only thing telling the encode to convert the audio,
|
||||
# so without this a Go Live stream with the timecode band AND Opus audio got -c:a copy and shipped Opus
|
||||
# inside an MP4 — legal, but editors choke on it.
|
||||
probe_reason="$reason"
|
||||
if (( live )); then
|
||||
[[ $verdict == skip ]] && continue
|
||||
[[ $verdict == ok ]] && verdict=remux # must leave .live/ either way
|
||||
@@ -173,12 +185,21 @@ PYEOF
|
||||
# scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
|
||||
vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
|
||||
(( deband )) && vf="crop=iw:ih-${band}:0:0,$vf" # drop the band rows first (band is even, so sizes stay even)
|
||||
fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
|
||||
fpsargs=()
|
||||
read -r fps_r fps_a < <(ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,avg_frame_rate -of csv=p=0 "$src" | tr ',' ' ')
|
||||
fpsnow=$(awk -v rfr="${fps_r:-0/1}" -v afr="${fps_a:-0/1}" 'BEGIN{
|
||||
split(rfr, a, "/"); split(afr, b, "/")
|
||||
r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
|
||||
if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
|
||||
awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
|
||||
# streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
|
||||
fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
|
||||
if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
|
||||
if (( live )) && [[ $reason != *audio=* ]]; then aargs=(-c:a copy); fi
|
||||
# Capping the rate means re-timing, which is by definition not passthrough. ffmpeg refuses the pair
|
||||
# outright ("One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode") and fails with
|
||||
# "Error opening output files: Invalid argument", so the cap has to win.
|
||||
(( ${#fpsargs[@]} )) && fpsmode=cfr
|
||||
if (( live )) && [[ $probe_reason != *audio=* ]]; then aargs=(-c:a copy); fi
|
||||
nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
|
||||
-map 0:v:0 -map '0:a?' \
|
||||
-c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \
|
||||
|
||||
@@ -18,6 +18,16 @@
|
||||
# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
|
||||
set -uo pipefail
|
||||
|
||||
# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
|
||||
# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
|
||||
# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
|
||||
if [[ -r /etc/vanguard/alerts.env ]]; then
|
||||
while IFS='=' read -r _k _v; do
|
||||
[[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
|
||||
[[ -n ${!_k:-} ]] || export "$_k=$_v"
|
||||
done < /etc/vanguard/alerts.env
|
||||
fi
|
||||
|
||||
ES="${ES_URL:-http://127.0.0.1:9200}"
|
||||
ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
|
||||
INDEX="${ALERT_INDEX:-vanguard-alerts}"
|
||||
@@ -38,15 +48,20 @@ state_set() {
|
||||
mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
|
||||
}
|
||||
|
||||
webhook=""
|
||||
[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE")
|
||||
# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like
|
||||
# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit.
|
||||
case "${webhook:-}" in
|
||||
"") ;;
|
||||
https://*) ;;
|
||||
*) vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;;
|
||||
esac
|
||||
# Delivery (and the mention handling) is shared with the deadman. If the helper is missing — a half-finished
|
||||
# deploy — fall back to posting without mentions rather than losing alerts entirely.
|
||||
if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
|
||||
. "$(dirname "$0")/vgnotify.sh"
|
||||
else
|
||||
vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
|
||||
vg_post() {
|
||||
local url; url=$(vg_webhook) || return 1
|
||||
curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
|
||||
--data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
|
||||
"$url"
|
||||
}
|
||||
fi
|
||||
webhook=$(vg_webhook 2>/dev/null || true)
|
||||
|
||||
now=$(date +%s)
|
||||
cursor=$(state_get cursor); cursor="${cursor:-0}"
|
||||
@@ -132,17 +147,12 @@ while IFS=$'\t' read -r millis rule level message; do
|
||||
state_set cursor "$millis"
|
||||
continue
|
||||
fi
|
||||
payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))')
|
||||
# curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000.
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \
|
||||
-d "$payload" "$webhook" || true)
|
||||
code="${code:-000}"
|
||||
if [[ $code == 2* ]]; then
|
||||
if vg_post "$level" "$text"; then
|
||||
sent=$((sent + 1)); state_set "sent:$rule" "$now"
|
||||
vglog info "event=delivered rule=\"$rule\" level=$level"
|
||||
vglog info "event=delivered rule=\"$rule\" level=$level mention=$([[ -n ${ALERT_MENTION:-} && ,${ALERT_MENTION_LEVELS:-error}, == *,$level,* ]] && echo yes || echo no)"
|
||||
else
|
||||
failed=$((failed + 1))
|
||||
vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2
|
||||
vglog error "event=delivery_failed rule=\"$rule\" level=$level" >&2
|
||||
fi
|
||||
state_set cursor "$millis"
|
||||
done <<<"$rows"
|
||||
|
||||
@@ -24,6 +24,16 @@
|
||||
# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
|
||||
set -uo pipefail
|
||||
|
||||
# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
|
||||
# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
|
||||
# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
|
||||
if [[ -r /etc/vanguard/alerts.env ]]; then
|
||||
while IFS='=' read -r _k _v; do
|
||||
[[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
|
||||
[[ -n ${!_k:-} ]] || export "$_k=$_v"
|
||||
done < /etc/vanguard/alerts.env
|
||||
fi
|
||||
|
||||
ES="${ES_URL:-http://127.0.0.1:9200}"
|
||||
# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
|
||||
# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
|
||||
@@ -130,26 +140,28 @@ was="$(cat "$STATE" 2>/dev/null || true)"
|
||||
printf '%s' "$now" > "$STATE"
|
||||
[[ $now == "$was" ]] && exit 0
|
||||
|
||||
webhook=""
|
||||
if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
|
||||
case "$webhook" in
|
||||
https://*) ;;
|
||||
"") webhook="" ;;
|
||||
*) echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
|
||||
esac
|
||||
# Delivery and mention handling are shared with vanguard-alerts.sh. A missing helper falls back to a plain
|
||||
# post: losing the mention is acceptable, losing the alert is not.
|
||||
if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
|
||||
. "$(dirname "$0")/vgnotify.sh"
|
||||
else
|
||||
vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
|
||||
vg_post() {
|
||||
local url; url=$(vg_webhook) || return 1
|
||||
curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
|
||||
--data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
|
||||
"$url"
|
||||
}
|
||||
fi
|
||||
webhook=$(vg_webhook 2>/dev/null || true)
|
||||
|
||||
post() {
|
||||
[[ -n $webhook ]] || return 1
|
||||
curl -s -m 15 -H 'Content-Type: application/json' \
|
||||
--data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
|
||||
"$webhook" >/dev/null && return 0
|
||||
echo "deadman: the webhook post failed" >&2
|
||||
return 1
|
||||
}
|
||||
# A deadman alarm is the one that must wake somebody: it means no other alert can get through. Recovery is
|
||||
# sent at info level so good news never pings.
|
||||
post() { vg_post "${2:-error}" "$1"; }
|
||||
|
||||
if [[ -z $now ]]; then
|
||||
echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
|
||||
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
|
||||
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." info || true
|
||||
exit 0
|
||||
fi
|
||||
|
||||
|
||||
74
bin/vgnotify.sh
Normal file
74
bin/vgnotify.sh
Normal file
@@ -0,0 +1,74 @@
|
||||
# Shared Discord delivery for bin/vanguard-alerts.sh and bin/vanguard-deadman.sh. Sourced, never executed.
|
||||
#
|
||||
# . /usr/local/bin/vgnotify.sh
|
||||
# vg_post error "disk is nearly full" # 0 = delivered, 1 = not (no webhook, bad URL, or HTTP failure)
|
||||
#
|
||||
# WHY MENTIONS NEED allowed_mentions
|
||||
# Putting <@&ROLE_ID> in a webhook's content is not enough on its own: Discord will render it but will not
|
||||
# reliably ping, because a webhook has no permission to mention roles unless the request says which mentions
|
||||
# are allowed. So every post sends an explicit allowed_mentions. It also sets parse: [] always, which means
|
||||
# an @everyone appearing inside an alert's text (a log line, a filename) can never ping the whole server.
|
||||
#
|
||||
# ALERT_MENTION is written for humans rather than for Discord's syntax:
|
||||
# ALERT_MENTION='role:1387247501387628585,role:1566623319027880086' -> pings those roles
|
||||
# ALERT_MENTION='user:123456789' -> pings that person
|
||||
# Raw '<@&123>' / '<@123>' is accepted too and passed through unchanged.
|
||||
#
|
||||
# ALERT_MENTION_LEVELS decides which alarms are worth a ping; the default is errors only, so warnings still
|
||||
# arrive in the channel without waking anyone.
|
||||
|
||||
VG_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
|
||||
VG_MENTION="${ALERT_MENTION:-}"
|
||||
VG_MENTION_LEVELS="${ALERT_MENTION_LEVELS:-error}"
|
||||
|
||||
# The webhook URL, or nothing. A value that is not https is treated as absent and complained about once:
|
||||
# posting to it would fail anyway, and silently "succeeding" would be worse.
|
||||
vg_webhook() {
|
||||
local url
|
||||
[[ -r $VG_WEBHOOK_FILE ]] || return 1
|
||||
url=$(tr -d '[:space:]' < "$VG_WEBHOOK_FILE" 2>/dev/null) || return 1
|
||||
case "$url" in
|
||||
https://*) printf '%s' "$url" ;;
|
||||
"") return 1 ;;
|
||||
*) echo "vgnotify: the configured webhook is not an https URL; ignoring it" >&2; return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
vg_post() {
|
||||
local level="$1" text="$2" url payload code
|
||||
url=$(vg_webhook) || return 1
|
||||
payload=$(VG_LEVEL="$level" VG_TEXT="$text" VG_MENTION="$VG_MENTION" VG_LEVELS="$VG_MENTION_LEVELS" python3 - <<'PY'
|
||||
import json, os, re
|
||||
level = os.environ.get("VG_LEVEL", "")
|
||||
text = os.environ.get("VG_TEXT", "")
|
||||
raw = os.environ.get("VG_MENTION", "").strip()
|
||||
levels = [l.strip() for l in os.environ.get("VG_LEVELS", "error").split(",") if l.strip()]
|
||||
|
||||
users, roles, parts = [], [], []
|
||||
for entry in [e.strip() for e in raw.replace(" ", ",").split(",") if e.strip()]:
|
||||
if entry.startswith("role:"):
|
||||
roles.append(entry[5:]); parts.append(f"<@&{entry[5:]}>")
|
||||
elif entry.startswith("user:"):
|
||||
users.append(entry[5:]); parts.append(f"<@{entry[5:]}>")
|
||||
else: # already Discord syntax
|
||||
parts.append(entry)
|
||||
roles += re.findall(r"<@&(\d+)>", entry)
|
||||
users += re.findall(r"<@!?(\d+)>", entry)
|
||||
|
||||
ping = bool(parts) and level in levels
|
||||
prefix = (" ".join(parts) + " ") if ping else ""
|
||||
# parse: [] unconditionally — an @everyone inside alert text must never ping the server.
|
||||
body = {"content": (prefix + text)[:1900],
|
||||
"allowed_mentions": {"parse": [],
|
||||
"users": sorted(set(users)) if ping else [],
|
||||
"roles": sorted(set(roles)) if ping else []}}
|
||||
print(json.dumps(body))
|
||||
PY
|
||||
)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST \
|
||||
-H 'Content-Type: application/json' -d "$payload" "$url" || true)
|
||||
case "${code:-000}" in
|
||||
2*) return 0 ;;
|
||||
*) echo "vgnotify: webhook post failed, HTTP ${code:-000}" >&2; return 1 ;;
|
||||
esac
|
||||
}
|
||||
Reference in New Issue
Block a user