Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
read, and allows retry/bin only for the admin groups; the path validation is the single thing
between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
them.
Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
verifies each step by reading it back.
Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
binary that is not there is a skip, not a failure.
Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
