techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Browse Source 
Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-04 21:25:43 -06:00
parent a9de09fa18
commit ca01516d42
18 changed files with 672 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
configure.py
View File

@@ -345,10 +345,23 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
# vanguard:diag
{diag_host} {{
encode zstd gzip
{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there
# is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published.
reverse_proxy 127.0.0.1:5601 {{
flush_interval -1
{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # Triage lives on this host, alongside the dashboards it belongs with. Both handles come BEFORE the
# catch-all below, and the Kibana proxy is wrapped in its own handle so the three are evaluated in the
# order written rather than by Caddy's directive ordering.
handle /stuck/api {{
reverse_proxy 127.0.0.1:8080 # dumpd: the list, and retry/bin for the admin groups
}}
handle_path /stuck* {{
root * /srv/stuck
try_files {{path}} /index.html
file_server # stuck-recording triage; this host is already media-admins only
}}
# Kibana IS the rest of this host: logs (Discover), dashboards and alerting in one app, served at the
# root so there is no path prefix to get wrong. Elasticsearch itself is never published.
handle {{
reverse_proxy 127.0.0.1:5601 {{
flush_interval -1
}}
}}
}}
"""
@@ -390,7 +403,14 @@ groups: {{http.request.header.X-Authentik-Groups}}"
reverse_proxy 127.0.0.1:8889
}}
handle_path /hls/* {{
reverse_proxy 127.0.0.1:8888
# handle_path strips /hls before proxying, so MediaMTX's own redirects come back root-relative and
# lose the prefix. It redirects every playlist request once, to ?cookieCheck=1, so without this the
# browser follows Location: /<user>/<Title>/index.m3u8 into the SPA catch-all below, receives
# index.html, and hls.js reports "manifestParsingError" — a web page where a manifest should be.
reverse_proxy 127.0.0.1:8888 {{
header_down Location ^/ /hls/
flush_interval -1 # low-latency HLS sends partial segments; do not buffer them
}}
}}
handle {{
root * /srv/live
@@ -709,23 +729,15 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"]) # script counters -> dumpd /metrics
# The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
# /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
"bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
if media:
run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
if Path("bin/vgmetrics.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband
if Path("bin/vgnotify.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgnotify.sh", "/usr/local/bin/"]) # sourced by the alerts timer and the deadman
# Everything bin/install.list names. Without this an --apply updated the containers but left STALE
# binaries in /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time.
for mode, path in bin_manifest():
run(["sudo", "install", "-m", mode, path, "/usr/local/bin/"])
if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
# The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
run(["sudo", "systemctl", "restart", "dumpd"])
@@ -741,6 +753,26 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
print(" run docker compose restart mediamtx between streams (cuts a live recording). Adding/removing")
print(" streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
def bin_manifest():
"""(mode, path) for everything that belongs in /usr/local/bin, from bin/install.list.
One list, shared with install.sh and deploy.sh. Three hand-kept copies had already drifted: a
push-to-deploy was installing five of the ten, so vgstuck.sh, both sourced helpers and both timer
scripts never reached the host unless somebody ran --apply by hand.
"""
out = []
try:
for line in Path("bin/install.list").read_text().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
mode, path = line.split(None, 1)
if Path(path).exists():
out.append((mode, path))
except FileNotFoundError:
pass
return out
def compose_cmd():
"""docker compose, with sudo only if this user cannot reach the daemon."""
if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0: