Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms
Triage - bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up. - The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate read, and allows retry/bin only for the admin groups; the path validation is the single thing between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside the streams tree. - The sweep logs event=parked once per run so Kibana can name stuck files rather than only count them. Landing - diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management. Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats' setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed. - /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and verifies each step by reading it back. Installers - bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh, configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which nothing did before. - check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node binary that is not there is a skip, not a failure. Fixes - Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned. - HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back root-relative and landed on the SPA catch-all. header_down rewrites Location. - Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately. - Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit. - Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -120,6 +120,7 @@
|
||||
<h2>Media admin</h2>
|
||||
<p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p>
|
||||
<div class="links">
|
||||
<a class="card" id="l-stuck" href="#"><b>Stuck recordings</b><span>never published — see why, retry or bin</span></a>
|
||||
<a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
|
||||
<a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a>
|
||||
<a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a>
|
||||
@@ -129,6 +130,7 @@
|
||||
<li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li>
|
||||
<li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li>
|
||||
<li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li>
|
||||
<li><a class="l-stuck-note" href="#">Stuck recordings</a> (on the diagnostics host, with the dashboards) lists anything that finished but never published, with the reason. Retry puts it back in the queue; binning keeps the same 3-day undo as a dumped night.</li>
|
||||
</ul>
|
||||
</details>
|
||||
</section>
|
||||
@@ -179,6 +181,8 @@ $('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = s
|
||||
$('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin';
|
||||
$('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/';
|
||||
$('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/';
|
||||
$('l-stuck').href = sub('diag') + '/stuck';
|
||||
document.querySelectorAll('a.l-stuck-note').forEach(a => a.href = sub('diag') + '/stuck');
|
||||
$('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
|
||||
(async () => {
|
||||
try {
|
||||
|
||||
Reference in New Issue
Block a user