Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms
Triage - bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up. - The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate read, and allows retry/bin only for the admin groups; the path validation is the single thing between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside the streams tree. - The sweep logs event=parked once per run so Kibana can name stuck files rather than only count them. Landing - diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management. Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats' setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed. - /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and verifies each step by reading it back. Installers - bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh, configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which nothing did before. - check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node binary that is not there is a skip, not a failure. Fixes - Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned. - HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back root-relative and landed on the SPA catch-all. header_down rewrites Location. - Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately. - Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit. - Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
15
deploy.sh
15
deploy.sh
@@ -46,20 +46,31 @@ apply_changes() {
|
||||
if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
|
||||
|
||||
# 1. host scripts + units
|
||||
if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi
|
||||
# bin/install.list, not a second hand-kept list: this one had drifted to five of ten files, so a push
|
||||
# never installed vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
|
||||
if has '^bin/'; then
|
||||
while read -r mode path; do
|
||||
[[ -f $path ]] && run install -m "$mode" "$path" /usr/local/bin/
|
||||
done < <(grep -vE '^[[:space:]]*(#|$)' bin/install.list)
|
||||
fi
|
||||
if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
|
||||
if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
|
||||
if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi
|
||||
if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi
|
||||
|
||||
# 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below
|
||||
if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/'; then
|
||||
if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/|^stuck/|^observability/kibana-setup\.py$'; then
|
||||
local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
|
||||
if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi
|
||||
local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
|
||||
[[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; }
|
||||
fi
|
||||
if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi
|
||||
# Beat and Kibana configs are bind-mounted, so a content change needs the container recreated. Nothing
|
||||
# handled this before: a push that changed filebeat.yml silently did nothing.
|
||||
if has '^observability/filebeat\.yml$'; then run docker compose up -d --force-recreate filebeat; fi
|
||||
if has '^observability/metricbeat\.yml$'; then run docker compose up -d --force-recreate metricbeat; fi
|
||||
if has '^observability/kibana\.yml$'; then run docker compose up -d --force-recreate kibana; fi
|
||||
|
||||
# 3. compose: recreate changed services, but never MediaMTX while live
|
||||
if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then
|
||||
|
||||
Reference in New Issue
Block a user