Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms
Triage - bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up. - The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate read, and allows retry/bin only for the admin groups; the path validation is the single thing between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside the streams tree. - The sweep logs event=parked once per run so Kibana can name stuck files rather than only count them. Landing - diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management. Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats' setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed. - /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and verifies each step by reading it back. Installers - bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh, configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which nothing did before. - check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node binary that is not there is a skip, not a failure. Fixes - Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned. - HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back root-relative and landed on the SPA catch-all. header_down rewrites Location. - Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately. - Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit. - Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -34,7 +34,11 @@ export async function captureWindow() {
|
||||
}
|
||||
}
|
||||
|
||||
export const MIN_KBPS = 6000
|
||||
// A FLOOR, not a target. It was 6000, which told Chrome never to encode below 6 Mbps: on an uplink that
|
||||
// could not sustain that, Chrome kept sending anyway, the network discarded the excess, and MediaMTX logged
|
||||
// "invalid FU-A packet (non-starting)" while every viewer saw blocky corruption. A floor this low still
|
||||
// skips the painfully slow cold ramp without disabling congestion control.
|
||||
export const MIN_KBPS = 1200
|
||||
export const START_KBPS = 9000
|
||||
|
||||
function preferCodecs(pc, kind = 'video') {
|
||||
@@ -54,7 +58,8 @@ function preferCodecs(pc, kind = 'video') {
|
||||
tx.setCodecPreferences([...high, ...base, ...vp9, ...rest])
|
||||
}
|
||||
|
||||
// Chrome honours x-google-* fmtp hints: skip the slow bandwidth ramp and never sit below MIN_KBPS.
|
||||
// Chrome honours x-google-* fmtp hints. start-bitrate skips the slow cold ramp, max-bitrate is the ceiling,
|
||||
// and min-bitrate is a floor that must stay LOW: a high floor defeats congestion control entirely.
|
||||
function mungeBitrates(sdp) {
|
||||
return sdp.replace(/^a=fmtp:(\d+) (.*)$/gm, (line, pt, params) =>
|
||||
/packetization-mode|profile-id|VP9/i.test(params) || /H264|VP9/i.test(sdp.match(new RegExp(`a=rtpmap:${pt} .*`))?.[0] || '')
|
||||
@@ -95,9 +100,10 @@ export async function publish({ stream, path, user, pass, onState }) {
|
||||
const vs = pc.getSenders().find(s => s.track?.kind === 'video')
|
||||
if (vs) {
|
||||
const p = vs.getParameters(); p.encodings = p.encodings?.length ? p.encodings : [{}]
|
||||
p.encodings[0].maxBitrate = MAX_KBPS * 1000
|
||||
p.encodings[0].maxBitrate = MAX_KBPS * 1000 // a ceiling; congestion control may go far below it
|
||||
p.encodings[0].maxFramerate = 60
|
||||
p.encodings[0].scaleResolutionDownBy = 1
|
||||
// No scaleResolutionDownBy: pinning it to 1 forbids the encoder from shedding resolution under
|
||||
// pressure, which leaves it nothing to give but packets it cannot deliver.
|
||||
p.degradationPreference = 'balanced'
|
||||
await vs.setParameters(p).catch(() => {})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user