Alarm mentions, targeted --set, and a logout that actually logs out

Mentions
- bin/vgnotify.sh is shared by the alert timer and the deadman. A webhook will not ping a role
  from <@&id> in the content alone: the request has to whitelist it, so every post now sends an
  explicit allowed_mentions. parse is always [] as well, so an @everyone appearing inside a log
  line or filename in an alert text can never ping the server.
- ALERT_MENTION is written as role:<id>/user:<id> rather than raw Discord syntax, and
  ALERT_MENTION_LEVELS keeps pings to errors by default so warnings arrive quietly. The deadman
  pings too, because its alarm means nothing else can get through, but its recovery is info level.
- Both scripts now read /etc/vanguard/alerts.env themselves instead of relying on systemd to
  inject it. Running one by hand previously behaved differently from the timer, which is exactly
  how "the mention is configured but never appears" happened.

configure.py --set
- Change one setting and apply only what it needs. --full re-prompts for every hostname and
  secret, which is far too blunt for pointing the alarms at a new channel, and mistyping a
  hostname there has broken TLS before.
- Every Discord credential and webhook is now one commented block at the top of .env, so there is
  one place to look, and --show prints the same five together with the secrets truncated.
- Each key applies only its own side effect: the webhook rewrites its root-only file, the deploy
  webhook rewrites deploy.env, the bot token recreates the bot container, log retention re-injects
  the ILM delete phase. LIVE_WEBHOOK_URL deliberately stops short, because MediaMTX only reads it
  at start and a recreate cuts a live recording.

Logout
- The proxy provider pointed at default-provider-invalidation-flow ("Logged out of application"),
  which ends only that application's session and leaves the SSO session alive, so /logout dropped
  the proxy cookie and forward-auth signed the user straight back in. It looked like a refresh.
  default-invalidation-flow ("Logout") is what Authentik's own button uses. --apply now enforces

  it had explicitly recommended the wrong flow.

Sweep
- The band-removal branch overwrote $reasocodec> marker, which is the
  only signal telling the encode to convert audio. A Go Live stream with both the band and Opus
  audio therefore got -c:a copy and shippeachable once Windows Go
  Live audio started working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 21:07:17 -06:00
parent b38d40ab0c
commit a9de09fa18
11 changed files with 371 additions and 46 deletions

View File

@@ -294,6 +294,14 @@ Discord notifier omitted, so nothing is half-configured and nothing fails to loa
value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
`use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
`.counter` and `.rate`. Look for the right one before concluding a metric is missing.
- **`-r` and a non-CFR `-fps_mode` are mutually exclusive in ffmpeg.** It refuses the pair with
"One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode" and then fails with
`Error opening output files: Invalid argument` — which looks like a path or permission problem and is not.
Capping the frame rate means re-timing, so the cap forces CFR and passthrough has to lose.
- **`avg_frame_rate` alone is not a frame rate.** A looped or fragmented recording reported 95 fps for a
30 fps source (`r_frame_rate=30/1`, `avg_frame_rate=3625/38`), which sent it down the fps-cap path and broke
the encode. `r_frame_rate` alone is no better: it is a huge nominal value on genuine VFR. The sweep takes
the **lower of the two**.
- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)

View File

@@ -69,6 +69,7 @@ works from ~/vanguard-stream without sudo.
./configure.py --full # first time: asks everything, then applies
./configure.py # later: add/remove/reset streamers and editors, then apply
./configure.py --show # print current users and the OBS settings to hand out
./configure.py --set K=V # change one setting (Discord tokens/webhooks, retention) without a full pass
Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
`Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
@@ -368,7 +369,8 @@ React + dockview page (`live/`) behind Authentik forward-auth: every active stre
audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
the optional "is live" Discord post (`LIVE_WEBHOOK_URL` in `.env`, asked by `configure.py --full`) and the
the optional "is live" Discord post (`LIVE_WEBHOOK_URL`, one of the Discord settings grouped at the top of
`.env`; set it with `./configure.py --set LIVE_WEBHOOK_URL=…`, which tells you MediaMTX needs a recreate) and the
**WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
@@ -416,9 +418,13 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
Logout: copyparty's logout link, the live portal's "Sign out" and the dock's "sign out" go through
`/outpost.goauthentik.io/sign_out`; `https://<any gated host>/logout` does the same from anywhere
(e.g. from the Syncthing UI, which has no logout of its own). Set the provider's **Invalidation flow** to
`default-provider-invalidation-flow` so that also ends the Authentik login: every logout then forces a full
Discord re-login on the next visit (and picks up group changes). Discord itself may approve silently if
you are still logged into Discord in that browser.
`default-invalidation-flow` — the one named **"Logout"**, the same flow Authentik's own logout button uses.
Do NOT use `default-provider-invalidation-flow`: despite the name it is called "Logged out of application"
and ends the session for that one application only, leaving the SSO session alive. With it, `/logout` drops
the proxy cookie, forward-auth asks Authentik again, Authentik still knows you, and you are signed straight
back in — which looks exactly like the page merely refreshing. With `default-invalidation-flow` every
logout ends the whole session and forces a full Discord re-login on the next visit (and so picks up group
changes). Discord itself may still approve silently if you remain logged into Discord in that browser.
5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them, and
`https://HOST/me` adds the `discord_id` claim decoded from `X-Authentik-Jwt` — the quickest check that the
`Vanguard profile` scope mapping is live. The raw token is deliberately never echoed: the browser does not

View File

@@ -106,6 +106,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
- Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
- The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
- **An encode that fails with `Error opening output files: Invalid argument` is usually contradictory flags, not the path.** ffmpeg rejects `-r` together with a non-CFR `-fps_mode`; the real message is one line earlier in its stderr, which the sweep now captures into the failure log. Found 2026-11-03 on the 1 November test recordings, which also exposed that `avg_frame_rate` can be pure fiction (95 fps reported for a 30 fps source) — the sweep now takes the lower of r_frame_rate and avg_frame_rate.
- A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
- MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
- **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
@@ -116,6 +117,10 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
- Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
- **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
- The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
- **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.
- Prefer changing a proxy provider through the ORM (`ak shell`) over the UI: saving it in the UI re-adds the managed property mappings every time.
- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.

View File

@@ -63,7 +63,14 @@ probe() {
afr=$(jq -r '[.streams[]|select(.codec_type=="video")][0].avg_frame_rate // "0/1"' <<<"$json")
acodecs=$(jq -r '[.streams[]|select(.codec_type=="audio")|.codec_name]|unique|join(",")' <<<"$json")
[[ $vcodec == none ]] && { echo "skip no-video"; return; }
local fps; fps=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$afr")
# Take the LOWER of r_frame_rate and avg_frame_rate. Either is unreliable alone: avg is garbage on a
# fragmented or looped recording (one reported 95 fps for a 30 fps source, which sent it down the fps-cap
# path and broke the encode), and r is a huge nominal value on genuine VFR. The lower of the two is the
# honest answer to "is this really above 60".
local fps; fps=$(awk -v rfr="$rfr" -v afr="$afr" 'BEGIN{
split(rfr, a, "/"); split(afr, b, "/")
r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
local reasons=()
[[ $vcodec == h264 ]] || reasons+=("codec=$vcodec")
[[ $pix == yuv420p ]] || reasons+=("pix=$pix")
@@ -113,6 +120,11 @@ while IFS= read -r -d '' src; do
[[ -e "$src" ]] || continue # dumped meanwhile
read -r verdict reason <<<"$(probe "$src")"
# Keep what the PROBE said: the deband branch below overwrites $reason with its own text, which silently
# loses the "audio=<codec>" marker. That marker is the only thing telling the encode to convert the audio,
# so without this a Go Live stream with the timecode band AND Opus audio got -c:a copy and shipped Opus
# inside an MP4 — legal, but editors choke on it.
probe_reason="$reason"
if (( live )); then
[[ $verdict == skip ]] && continue
[[ $verdict == ok ]] && verdict=remux # must leave .live/ either way
@@ -173,12 +185,21 @@ PYEOF
# scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
(( deband )) && vf="crop=iw:ih-${band}:0:0,$vf" # drop the band rows first (band is even, so sizes stay even)
fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
fpsargs=()
read -r fps_r fps_a < <(ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,avg_frame_rate -of csv=p=0 "$src" | tr ',' ' ')
fpsnow=$(awk -v rfr="${fps_r:-0/1}" -v afr="${fps_a:-0/1}" 'BEGIN{
split(rfr, a, "/"); split(afr, b, "/")
r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
# streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
if (( live )) && [[ $reason != *audio=* ]]; then aargs=(-c:a copy); fi
# Capping the rate means re-timing, which is by definition not passthrough. ffmpeg refuses the pair
# outright ("One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode") and fails with
# "Error opening output files: Invalid argument", so the cap has to win.
(( ${#fpsargs[@]} )) && fpsmode=cfr
if (( live )) && [[ $probe_reason != *audio=* ]]; then aargs=(-c:a copy); fi
nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
-map 0:v:0 -map '0:a?' \
-c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \

View File

@@ -18,6 +18,16 @@
# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
set -uo pipefail
# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
if [[ -r /etc/vanguard/alerts.env ]]; then
while IFS='=' read -r _k _v; do
[[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
[[ -n ${!_k:-} ]] || export "$_k=$_v"
done < /etc/vanguard/alerts.env
fi
ES="${ES_URL:-http://127.0.0.1:9200}"
ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
INDEX="${ALERT_INDEX:-vanguard-alerts}"
@@ -38,15 +48,20 @@ state_set() {
mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
}
webhook=""
[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE")
# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like
# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit.
case "${webhook:-}" in
"") ;;
https://*) ;;
*) vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;;
esac
# Delivery (and the mention handling) is shared with the deadman. If the helper is missing — a half-finished
# deploy — fall back to posting without mentions rather than losing alerts entirely.
if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
. "$(dirname "$0")/vgnotify.sh"
else
vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
vg_post() {
local url; url=$(vg_webhook) || return 1
curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
--data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
"$url"
}
fi
webhook=$(vg_webhook 2>/dev/null || true)
now=$(date +%s)
cursor=$(state_get cursor); cursor="${cursor:-0}"
@@ -132,17 +147,12 @@ while IFS=$'\t' read -r millis rule level message; do
state_set cursor "$millis"
continue
fi
payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))')
# curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000.
code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \
-d "$payload" "$webhook" || true)
code="${code:-000}"
if [[ $code == 2* ]]; then
if vg_post "$level" "$text"; then
sent=$((sent + 1)); state_set "sent:$rule" "$now"
vglog info "event=delivered rule=\"$rule\" level=$level"
vglog info "event=delivered rule=\"$rule\" level=$level mention=$([[ -n ${ALERT_MENTION:-} && ,${ALERT_MENTION_LEVELS:-error}, == *,$level,* ]] && echo yes || echo no)"
else
failed=$((failed + 1))
vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2
vglog error "event=delivery_failed rule=\"$rule\" level=$level" >&2
fi
state_set cursor "$millis"
done <<<"$rows"

View File

@@ -24,6 +24,16 @@
# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
set -uo pipefail
# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
if [[ -r /etc/vanguard/alerts.env ]]; then
while IFS='=' read -r _k _v; do
[[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
[[ -n ${!_k:-} ]] || export "$_k=$_v"
done < /etc/vanguard/alerts.env
fi
ES="${ES_URL:-http://127.0.0.1:9200}"
# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
@@ -130,26 +140,28 @@ was="$(cat "$STATE" 2>/dev/null || true)"
printf '%s' "$now" > "$STATE"
[[ $now == "$was" ]] && exit 0
webhook=""
if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
case "$webhook" in
https://*) ;;
"") webhook="" ;;
*) echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
esac
post() {
[[ -n $webhook ]] || return 1
curl -s -m 15 -H 'Content-Type: application/json' \
--data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
"$webhook" >/dev/null && return 0
echo "deadman: the webhook post failed" >&2
return 1
# Delivery and mention handling are shared with vanguard-alerts.sh. A missing helper falls back to a plain
# post: losing the mention is acceptable, losing the alert is not.
if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
. "$(dirname "$0")/vgnotify.sh"
else
vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
vg_post() {
local url; url=$(vg_webhook) || return 1
curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
--data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
"$url"
}
fi
webhook=$(vg_webhook 2>/dev/null || true)
# A deadman alarm is the one that must wake somebody: it means no other alert can get through. Recovery is
# sent at info level so good news never pings.
post() { vg_post "${2:-error}" "$1"; }
if [[ -z $now ]]; then
echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." info || true
exit 0
fi

74
bin/vgnotify.sh Normal file
View File

@@ -0,0 +1,74 @@
# Shared Discord delivery for bin/vanguard-alerts.sh and bin/vanguard-deadman.sh. Sourced, never executed.
#
# . /usr/local/bin/vgnotify.sh
# vg_post error "disk is nearly full" # 0 = delivered, 1 = not (no webhook, bad URL, or HTTP failure)
#
# WHY MENTIONS NEED allowed_mentions
# Putting <@&ROLE_ID> in a webhook's content is not enough on its own: Discord will render it but will not
# reliably ping, because a webhook has no permission to mention roles unless the request says which mentions
# are allowed. So every post sends an explicit allowed_mentions. It also sets parse: [] always, which means
# an @everyone appearing inside an alert's text (a log line, a filename) can never ping the whole server.
#
# ALERT_MENTION is written for humans rather than for Discord's syntax:
# ALERT_MENTION='role:1387247501387628585,role:1566623319027880086' -> pings those roles
# ALERT_MENTION='user:123456789' -> pings that person
# Raw '<@&123>' / '<@123>' is accepted too and passed through unchanged.
#
# ALERT_MENTION_LEVELS decides which alarms are worth a ping; the default is errors only, so warnings still
# arrive in the channel without waking anyone.
VG_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
VG_MENTION="${ALERT_MENTION:-}"
VG_MENTION_LEVELS="${ALERT_MENTION_LEVELS:-error}"
# The webhook URL, or nothing. A value that is not https is treated as absent and complained about once:
# posting to it would fail anyway, and silently "succeeding" would be worse.
vg_webhook() {
local url
[[ -r $VG_WEBHOOK_FILE ]] || return 1
url=$(tr -d '[:space:]' < "$VG_WEBHOOK_FILE" 2>/dev/null) || return 1
case "$url" in
https://*) printf '%s' "$url" ;;
"") return 1 ;;
*) echo "vgnotify: the configured webhook is not an https URL; ignoring it" >&2; return 1 ;;
esac
}
vg_post() {
local level="$1" text="$2" url payload code
url=$(vg_webhook) || return 1
payload=$(VG_LEVEL="$level" VG_TEXT="$text" VG_MENTION="$VG_MENTION" VG_LEVELS="$VG_MENTION_LEVELS" python3 - <<'PY'
import json, os, re
level = os.environ.get("VG_LEVEL", "")
text = os.environ.get("VG_TEXT", "")
raw = os.environ.get("VG_MENTION", "").strip()
levels = [l.strip() for l in os.environ.get("VG_LEVELS", "error").split(",") if l.strip()]
users, roles, parts = [], [], []
for entry in [e.strip() for e in raw.replace(" ", ",").split(",") if e.strip()]:
if entry.startswith("role:"):
roles.append(entry[5:]); parts.append(f"<@&{entry[5:]}>")
elif entry.startswith("user:"):
users.append(entry[5:]); parts.append(f"<@{entry[5:]}>")
else: # already Discord syntax
parts.append(entry)
roles += re.findall(r"<@&(\d+)>", entry)
users += re.findall(r"<@!?(\d+)>", entry)
ping = bool(parts) and level in levels
prefix = (" ".join(parts) + " ") if ping else ""
# parse: [] unconditionally — an @everyone inside alert text must never ping the server.
body = {"content": (prefix + text)[:1900],
"allowed_mentions": {"parse": [],
"users": sorted(set(users)) if ping else [],
"roles": sorted(set(roles)) if ping else []}}
print(json.dumps(body))
PY
)
code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST \
-H 'Content-Type: application/json' -d "$payload" "$url" || true)
case "${code:-000}" in
2*) return 0 ;;
*) echo "vgnotify: webhook post failed, HTTP ${code:-000}" >&2; return 1 ;;
esac
}

View File

@@ -4,6 +4,9 @@
./configure.py --full first-time setup: asks every question (current values as defaults)
./configure.py manage: skips what is already set; add/remove/reset streamers & editors
./configure.py --show print current streamers, editors, domain and the OBS settings
./configure.py --set K=V change ONE setting and apply only what it needs (Discord tokens and webhooks,
retention, alert tuning). Run it with no arguments to list what is settable.
Everything else regenerates a config or a container, so it needs --apply.
./configure.py --apply regenerate everything from the values already on disk and apply (non-interactive;
used by deploy.sh). Fails if the main hostname was never set.
./configure.py --hosts AUTH WATCH [GIT [BOT [LIVE [FILES [SYNC [CLIPS [DIAG]]]]]]]
@@ -55,6 +58,115 @@ def read_hosts():
def read_domain(): return read_hosts().get("main", "")
# Every Discord credential and webhook, in one block, in this order, with these comments written into .env.
# They used to be scattered through the file in the order they were added, so "where does the alarm webhook
# go" had no obvious answer. read_env ignores any line starting with #, so the comments are free.
DISCORD_KEYS = (
("DISCORD_BOT_TOKEN", "the bot's own token (Discord dev portal -> Bot). The bot's CI also sets this from Gitea"),
("DISCORD_GUILD_ID", "the server the bot and the Authentik membership policies work against"),
("ALERT_WEBHOOK_URL", "webhook: alarms (vanguard-alerts.timer) and the deadman"),
("LIVE_WEBHOOK_URL", "webhook: 'a stream went live' (MediaMTX runOnReady)"),
("DEPLOY_NOTIFY_WEBHOOK_URL", "webhook: deploy results (deploy.sh)"),
)
DEPLOY_ENV_KEYS = ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL",
"BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")
def render_env(env):
""".env text with the Discord block first and commented, everything else after it."""
named = dict(DISCORD_KEYS)
out = ["# ---- Discord: tokens and webhooks ----\n",
"# change any of these with: ./configure.py --set KEY=VALUE\n"]
for key, why in DISCORD_KEYS:
out.append(f"# {why}\n{key}={env.get(key, '')}\n")
out.append("\n# ---- everything else ----\n")
out += [f"{k}={v}\n" for k, v in env.items() if k not in named]
return "".join(out)
def write_deploy_env(env=None):
"""/etc/vanguard/deploy.env — read by dumpd's /deploy webhook and by deploy.sh."""
env = env or read_env()
Path("deploy.env").write_text("".join(f"{k}={env.get(k, '')}\n" for k in DEPLOY_ENV_KEYS)
+ f"DEPLOY_REPO_DIR={Path.cwd()}\n")
os.chmod("deploy.env", 0o600)
run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"])
Path("deploy.env").unlink(missing_ok=True)
# Settings that can be changed on their own, with the one side effect each needs. Everything else has to go
# through --apply or --full because it regenerates a config file or a container: DIAG_REQUIRE_GROUP needs the
# Caddyfile rewritten and reloaded, DIAG_KIBANA_ROLES needs the Elasticsearch user changed and Kibana
# restarted, hostnames need the whole pass. Keeping this list short is the point.
SETTABLE = {
"DISCORD_BOT_TOKEN": "the bot's token (restarts the bot container)",
"DISCORD_GUILD_ID": "the Discord server id (restarts the bot container)",
"ALERT_WEBHOOK_URL": "Discord webhook for alarms and the deadman (blank = journal only)",
"LIVE_WEBHOOK_URL": "Discord webhook for 'went live' (needs a MediaMTX recreate — NOT done for you)",
"DEPLOY_NOTIFY_WEBHOOK_URL": "Discord webhook for deploy results",
"ALERT_RETENTION_DAYS": "how long delivered alerts stay queryable in vanguard-alerts",
"ALERT_SUPPRESS_S": "per-rule repeat suppression in the delivery timer, seconds",
"ALERT_MENTION": "who an alarm pings, e.g. role:123,role:456 or user:789 (blank = nobody)",
"ALERT_MENTION_LEVELS": "which levels ping, comma separated (default error; add warn to widen)",
"HEARTBEAT_MAX_S": "how stale the alerting heartbeat may get before the deadman alarms, seconds",
"LOG_RETENTION_DAYS": "ILM delete age for filebeat-*/metricbeat-* (re-applied immediately)",
}
def set_env(updates):
"""Change individual .env keys in place and apply only what each one needs.
--full re-prompts for every hostname and secret, which is far too blunt for "point the alarms at a new
Discord channel" — mistyping a hostname there has broken TLS before. This writes the keys, rewrites the
files that derive from them, and touches nothing else.
"""
env = read_env()
if not env:
print("no .env here — run ./configure.py --full first"); sys.exit(1)
for key, value in updates.items():
if key not in SETTABLE:
print(f"{key} is not settable on its own. Settable keys:")
for name, why in SETTABLE.items():
print(f" {name:22} {why}")
print("Anything else regenerates a config or a container: use --apply (or --full for hostnames).")
sys.exit(1)
env[key] = value
Path(".env").write_text(render_env(env))
os.chmod(".env", 0o600)
for key, value in updates.items():
print(f" {key}={value if key != 'ALERT_WEBHOOK_URL' else (value[:40] + '…' if value else '(cleared)')}")
# The webhook is a credential for that channel, so it stays in its own root-only file.
if "ALERT_WEBHOOK_URL" in updates:
Path("alert-webhook").write_text((env.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
Path("alert-webhook").unlink(missing_ok=True)
print(" wrote /etc/vanguard/alert-webhook")
# Always, not only when one of its keys changed: it is two file writes, it is idempotent, and making it
# conditional means the file is missing on any system where the first --set happened to be the webhook.
# A systemd timer never reads the repo's .env, so this file is the only way those knobs reach the scripts.
write_alerts_env(env)
if "DEPLOY_NOTIFY_WEBHOOK_URL" in updates:
write_deploy_env(env); print(" wrote /etc/vanguard/deploy.env")
if "LOG_RETENTION_DAYS" in updates:
es_security(compose_cmd()) # re-injects the ILM delete phase at the new age
# The bot reads these from its container environment, so they only take effect on recreate. Its own CI
# does the same thing when it installs a bundle, so a restart here is nothing unusual for the bot.
if any(k in updates for k in ("DISCORD_BOT_TOKEN", "DISCORD_GUILD_ID")):
run(compose_cmd() + ["up", "-d", "discord-bot"]); print(" discord-bot recreated with the new settings")
# MediaMTX reads LIVE_WEBHOOK_URL from ITS environment too, but recreating MediaMTX cuts whatever is
# recording right now. That is never worth doing automatically, so it is left to a human and a quiet moment.
if "LIVE_WEBHOOK_URL" in updates:
print(" NOTE: MediaMTX still has the old value. It only reads this on start, and a restart cuts a")
print(" live recording, so run this yourself between streams: docker compose up -d mediamtx")
def write_alerts_env(env=None):
"""/etc/vanguard/alerts.env — read by vanguard-alerts.service and vanguard-deadman.service."""
env = env or read_env()
keys = ("ALERT_INDEX", "ALERT_RETENTION_DAYS", "ALERT_SUPPRESS_S", "HEARTBEAT_MAX_S",
"ALERT_MENTION", "ALERT_MENTION_LEVELS") # what the two scripts read
Path("alerts.env").write_text("".join(f"{k}={env[k]}\n" for k in keys if env.get(k)))
os.chmod("alerts.env", 0o644)
run(["sudo", "install", "-m", "0644", "-o", "root", "-g", "root", "alerts.env", "/etc/vanguard/alerts.env"])
Path("alerts.env").unlink(missing_ok=True)
print(" wrote /etc/vanguard/alerts.env")
def read_env():
env = {}
try:
@@ -408,10 +520,12 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
env.setdefault("ALERT_INDEX", "vanguard-alerts") # the alert bus: Kibana writes, vanguard-alerts.sh delivers
env.setdefault("LOG_RETENTION_DAYS", "30") # ILM delete phase for filebeat-*/metricbeat-*
env.setdefault("ALERT_RETENTION_DAYS", "90") # how long delivered alerts stay queryable
env.setdefault("ALERT_MENTION", "") # role:<id>/user:<id> list an alarm pings
env.setdefault("ALERT_MENTION_LEVELS", "error") # only errors ping; warnings arrive quietly
env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot
env.setdefault("DISCORD_GUILD_ID", "")
env["WATCH_HOST"] = watch_host or "watch.example.com"
Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items()))
Path(".env").write_text(render_env(env))
os.chmod(".env", 0o600)
def docker_gateway():
@@ -573,13 +687,13 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
run(["sudo", "install", "-m", "0600", "stream-token.secret", "/etc/vanguard/stream-token.secret"])
run(["sudo", "install", "-m", "0644", "syncthing/stignore", "/etc/vanguard/stignore"]) # dumpd copies it into new clips folders
e = read_env()
Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
write_deploy_env(e) # one copy of the key list, shared with --set
Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
# URL is a credential for that channel, so it is not world-readable.
# 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be
# group-readable by nobody(65534) for the Alertmanager container, which no longer exists.
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh
write_alerts_env(e) # the timers' own knobs; a systemd timer never reads .env
if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
run(["sudo", "install", "-m", "0644", "systemd/deploy-pending.service", "systemd/deploy-pending.timer", "/etc/systemd/system/"])
@@ -603,6 +717,8 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
if Path("bin/vgmetrics.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband
if Path("bin/vgnotify.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgnotify.sh", "/usr/local/bin/"]) # sourced by the alerts timer and the deadman
if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
@@ -752,11 +868,60 @@ def es_security(dc):
run(dc + ["restart", "kibana", "filebeat", "metricbeat"])
print(" elasticsearch: netrc written for the deadman; kibana + beats restarted")
def ak_shell(dc, script, label):
"""Run a short script inside authentik-server's Django shell and print only its own output.
`ak shell` is an interactive console: it reads stdin line by line and rejects indented blocks, so the
script goes in as a file and is executed by one unindented statement. Authentik also prints a wall of
JSON boot logs to the same stream, which is why the output is filtered.
"""
if DRY:
print(f" (dry) authentik: {label}"); return
path = Path("ak-fixup.py")
path.write_text(script)
try:
if subprocess.call(dc + ["cp", str(path), "authentik-server:/tmp/ak-fixup.py"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) != 0:
print(f" authentik: cannot reach the container — {label} skipped"); return
result = subprocess.run(dc + ["exec", "-T", "authentik-server", "ak", "shell",
"-c", "exec(open('/tmp/ak-fixup.py').read())"],
capture_output=True, text=True)
for line in (result.stdout + result.stderr).splitlines():
if line.startswith(("{", "#", ">>>", "...")) or not line.strip():
continue
print(" ", line.rstrip())
except Exception as error:
print(f" authentik: {label} skipped ({error})")
finally:
path.unlink(missing_ok=True)
# Proxy-provider state the Authentik UI can quietly put back. Enforced on every apply so a save in the UI
# cannot leave logout half-working for weeks without anyone noticing.
AK_LOGOUT_FIXUP = """
from authentik.providers.proxy.models import ProxyProvider
from authentik.flows.models import Flow
target = Flow.objects.filter(slug="default-invalidation-flow").first()
if target is None:
print("authentik: no default-invalidation-flow found; logout flow left alone")
else:
for provider in ProxyProvider.objects.all():
before = getattr(provider.invalidation_flow, "slug", None)
if before == target.slug:
print(f"authentik: {provider.name!r} logout flow ok ({before})")
else:
provider.invalidation_flow = target
provider.save(update_fields=["invalidation_flow"])
print(f"authentik: {provider.name!r} logout flow {before} -> {target.slug}")
"""
def post_start_fixups(dc):
"""Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin."""
env = read_env()
print("== post-start fixups (idempotent)")
es_security(dc) # before anything else: Kibana and the Beats cannot authenticate until this runs
# /logout must end the whole Authentik session, not just the application's. The provider-scoped flow
# ("Logged out of application") leaves the SSO session alive, so logout looks like a page refresh.
ak_shell(dc, AK_LOGOUT_FIXUP, "proxy provider logout flow")
# Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it
cfg = Path("/srv/media/.syncthing/config.xml")
if not DRY and cfg.exists():
@@ -835,8 +1000,16 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
print(f"== authentik: https://{auth_host}/ akadmin password = AUTHENTIK_BOOTSTRAP_PASSWORD in .env (set on first start; change it in the UI)" if auth_host else "== authentik: (not set — run --full)")
print(f"== jellyfin : https://{watch_host}/" if watch_host else "== jellyfin : (not set — run --full)")
env = read_env()
tok = env.get("DISCORD_BOT_TOKEN", "")
print(f"== discord : guild {env.get('DISCORD_GUILD_ID') or '(not set)'}; bot token {'set (' + tok[:6] + '…)' if tok else '(not set)'}")
print("== discord : (./configure.py --set KEY=VALUE)")
for key, why in DISCORD_KEYS:
value = env.get(key, "")
if not value:
shown = "(not set)"
elif key == "DISCORD_GUILD_ID":
shown = value
else: # tokens and webhook URLs are credentials: never print them whole
shown = value[:34] + "…" if len(value) > 34 else value[:8] + "…"
print(f" {key:26} {shown:40} {why}")
print(f"== mariadb : {domain or 'HOST'}:3306 (public, root local-only) root pass {env.get('MARIADB_ROOT_PASSWORD','(run apply)')}")
print(f" bot ci : Gitea -> VanguardBot -> Settings -> Actions -> Secrets: BOT_DEPLOY_SECRET={env.get('BOT_DEPLOY_SECRET','(run apply)')}")
print(f" pushes to main install themselves (BOT_DEPLOY_AUTO={env.get('BOT_DEPLOY_AUTO','1')}); by hand: ./deployBotBundle.sh <bundle>")
@@ -918,7 +1091,7 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
Path(".env").write_text(f"APEX_HOST={apex}\nAPEX_REDIRECT={apex_to}\n"); os.chmod(".env", 0o600)
if alert_webhook != env0.get("ALERT_WEBHOOK_URL", ""):
env = read_env(); env["ALERT_WEBHOOK_URL"] = alert_webhook
Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items())); os.chmod(".env", 0o600)
Path(".env").write_text(render_env(env)); os.chmod(".env", 0o600)
return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host
def ask_discord():
@@ -951,6 +1124,15 @@ def main():
streamers, accounts = read_streamers(), read_accounts()
admin_pw = accounts.pop("admin", "")
editors = accounts
if mode == "--set":
pairs = [a for a in sys.argv[2:] if "=" in a]
if not pairs:
print("usage: configure.py --set KEY=VALUE [KEY=VALUE ...]\n\nSettable on their own:")
for name, why in SETTABLE.items():
print(f" {name:22} {why}")
print('\nexample: ./configure.py --set ALERT_WEBHOOK_URL=https://discord.com/api/webhooks/…')
sys.exit(1)
set_env(dict(p.split("=", 1) for p in pairs)); return
if mode == "--es-security":
# Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an
# --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but

View File

@@ -58,6 +58,7 @@ done
echo "== host scripts + units"
install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
install -m 0644 bin/vgmetrics.sh /usr/local/bin/ # sourced by the sweep/vgdeband; counters for dumpd's /metrics
install -m 0644 bin/vgnotify.sh /usr/local/bin/ # sourced by the alerts timer + deadman; Discord posts and mentions
install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot
install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it
install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord

View File

@@ -4,6 +4,9 @@ After=docker.service
[Service]
Type=oneshot
# Written by configure.py from .env (ALERT_RETENTION_DAYS, ALERT_SUPPRESS_S, ALERT_INDEX). A timer never reads the repo's .env, so without
# this the tunables were settable in name only. The leading - means "carry on if it is absent".
EnvironmentFile=-/etc/vanguard/alerts.env
ExecStart=/usr/local/bin/vanguard-alerts.sh
# Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts
# index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md.

View File

@@ -4,4 +4,7 @@ After=docker.service
[Service]
Type=oneshot
# Written by configure.py from .env (HEARTBEAT_MAX_S, ALERT_INDEX). A timer never reads the repo's .env, so without
# this the tunables were settable in name only. The leading - means "carry on if it is absent".
EnvironmentFile=-/etc/vanguard/alerts.env
ExecStart=/usr/local/bin/vanguard-deadman.sh