techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Browse Source 
Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-04 21:25:43 -06:00
parent a9de09fa18
commit ca01516d42
18 changed files with 672 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

144
observability/kibana-setup.py
View File

@@ -4,6 +4,9 @@
connector "Vanguard alert index" — the .index connector, writing into the vanguard-alerts index
rule "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
data view "Vanguard logs + metrics" — filebeat-*,metricbeat-* so Discover and Lens have one source
landing "Vanguard ops" dashboard — our own links, and defaultRoute so diag.BASE opens on it
Run one step on its own by naming it: kibana-setup.py landing (also: connector, rule, dataview, all)
WHY AN INDEX AND NOT A WEBHOOK
On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
@@ -38,11 +41,25 @@ delivery script recognises the heartbeat and keeps it out of Discord. Use "On st
for real alarms; only the heartbeat wants to fire every interval.
"""
import json, os, time, urllib.error, urllib.request
from pathlib import Path # env_hosts() reads the stack's .env
import sys
STEPS = ("connector", "rule", "dataview", "landing")
WANTED = sys.argv[1] if len(sys.argv) > 1 else "all"
if WANTED not in STEPS + ("all",):
raise SystemExit(f"unknown step {WANTED!r}; pick one of: {', '.join(STEPS)}, all")
def wanted(step):
return WANTED in ("all", step)
KIBANA = "http://127.0.0.1:5601"
ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
"Authorization": "ApiKey " + os.environ["KEY"]}
"Authorization": "ApiKey " + os.environ["KEY"],
# Kibana 9 ships with server.restrictInternalApis on. /api/kibana/settings (defaultRoute) is an
# internal API and refuses without this header: "exists but is not available with the current
# configuration". Sending it on every call is harmless; loosening the restriction is not.
"x-elastic-internal-origin": "Kibana"}
def wait_for_kibana(seconds=300):
"""Kibana has to be *available*, not merely listening.
@@ -87,26 +104,32 @@ wait_for_kibana()
# 1. the alert bus connector
CONNECTOR_NAME = "Vanguard alert index"
existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
if CONNECTOR_NAME in existing:
connector_id = None
if wanted("connector") or wanted("rule"):
existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
if CONNECTOR_NAME in existing:
connector_id = existing[CONNECTOR_NAME]
print("connector already present:", connector_id)
else:
connector_id = req("POST", "/api/actions/connector", {
elif not wanted("connector"):
raise SystemExit(f"connector {CONNECTOR_NAME!r} does not exist yet — run the connector step first")
else:
connector_id = req("POST", "/api/actions/connector", {
"name": CONNECTOR_NAME,
"connector_type_id": ".index",
# executionTimeField stamps every document, so a rule body never has to get the timestamp right.
"config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
"secrets": {}})["id"]
print("connector created:", connector_id, "->", ALERT_INDEX)
print("connector created:", connector_id, "->", ALERT_INDEX)
# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
# harmless and meant the data view was never created on any system where the rule already existed, which
# is every re-run.
RULE_NAME = "Vanguard alerting heartbeat"
existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
existing_rule = None if not wanted("rule") else next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
if r["name"] == RULE_NAME), None)
if existing_rule:
if not wanted("rule"):
pass
elif existing_rule:
print("rule already present:", existing_rule["id"],
"enabled" if existing_rule.get("enabled") else "DISABLED")
else:
@@ -152,10 +175,111 @@ else:
# detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
# metrics, which is the whole reason they share a cluster.
VIEW_TITLE = "filebeat-*,metricbeat-*"
views = req("GET", "/api/data_views").get("data_view", [])
if any(v.get("title") == VIEW_TITLE for v in views):
views = req("GET", "/api/data_views").get("data_view", []) if wanted("dataview") else []
if not wanted("dataview"):
pass
elif any(v.get("title") == VIEW_TITLE for v in views):
print("data view already present:", VIEW_TITLE)
else:
view = req("POST", "/api/data_views/data_view", {
"data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)
# 4. the landing page. Kibana stays at the root of the diag host; this just makes arriving there open OUR
# dashboard instead of Kibana's home. A markdown panel holds the links, and `defaultRoute` is the
# advanced setting that does the redirect. That is the whole trick, and it avoids moving Kibana under a
# /kibana prefix — which would have meant touching both Beats' setup.kibana, this script, the alarm
# script and the deadman's status check, with a silent crash-loop waiting for whichever one was missed.
DASHBOARD_ID = "vanguard-ops"
def env_hosts():
"""Hostnames out of the stack's .env, so the links are real rather than examples."""
values = {}
for candidate in (Path(".env"), Path("/home/debian/vanguard-stream/.env")):
if candidate.is_file():
for line in candidate.read_text().splitlines():
if "=" in line and not line.startswith("#"):
key, value = line.split("=", 1)
values[key.strip()] = value.strip()
break
return values
def landing_markdown(hosts):
def url(key, path="/"):
host = hosts.get(key)
return f"https://{host}{path}" if host else None
rows = [
("Stuck recordings", "/stuck", "finished but never published — why, and retry or bin"),
("Live portal", url("LIVE_HOST"), "watch every active stream"),
("OBS dock", url("LIVE_HOST", "/dock/"), "go live, titles, dump tonight"),
("Files", url("FILES_HOST"), "browse, download, upload finals"),
("Jellyfin", url("WATCH_HOST"), "recordings by streamer and season"),
("Gateway", url("MAIN_HOST"), "role-aware links and how-tos"),
("Gitea", url("GIT_HOST"), "code and CI"),
("Bot", url("BOT_HOST"), "the Discord bot's own pages"),
("Syncthing", url("SYNC_HOST"), "clip intake, device pairing"),
("Authentik", url("AUTH_HOST", "/if/admin/"), "users, groups, policies"),
]
lines = ["### Vanguard", "", "| | |", "|---|---|"]
for name, link, why in rows:
if link:
lines.append(f"| **[{name}]({link})** | {why} |")
lines += [
"", "### In here", "",
"| | |", "|---|---|",
"| **[Discover](/app/discover)** | search every log line; filter on `service.name` or `vanguard.category` |",
"| **[Dashboards](/app/dashboards)** | the Metricbeat system and Docker dashboards |",
"| **[Stack Rules](/app/management/insightsAndAlerting/triggersActions/rules)** | the six alarms, and whether each is firing |",
"| **[Index Management](/app/management/data/index_management/indices)** | index sizes and ILM |",
"",
"_Alarms post to Discord through the `vanguard-alerts` index, not from Kibana directly: a basic "
"licence has no webhook connector. The host deadman watches that whole path from outside._",
]
return "\n".join(lines)
if wanted("landing"):
hosts = env_hosts()
panel = {
"type": "visualization",
"panelIndex": "1",
"gridData": {"x": 0, "y": 0, "w": 24, "h": 18, "i": "1"},
"embeddableConfig": {
"savedVis": {
"id": "", "title": "", "description": "", "type": "markdown",
"params": {"fontSize": 12, "openLinksInNewTab": False,
"markdown": landing_markdown(hosts)},
"uiState": {},
"data": {"aggs": [], "searchSource": {"query": {"query": "", "language": "kuery"},
"filter": []}},
},
"enhancements": {},
},
}
saved = req("POST", f"/api/saved_objects/dashboard/{DASHBOARD_ID}?overwrite=true", {
"attributes": {
"title": "Vanguard ops",
"description": "Landing page for the diagnostics host. Edit freely: add metric panels beside the links.",
"panelsJSON": json.dumps([panel]),
"optionsJSON": json.dumps({"hidePanelTitles": True, "useMargins": True,
"syncColors": False, "syncCursor": True, "syncTooltips": False}),
"timeRestore": False,
"kibanaSavedObjectMeta": {
"searchSourceJSON": json.dumps({"query": {"query": "", "language": "kuery"}, "filter": []})},
}})
# Prove it rather than assume it: read the dashboard back and count the panels Kibana actually stored.
stored = req("GET", f"/api/saved_objects/dashboard/{DASHBOARD_ID}")
panels = json.loads(stored.get("attributes", {}).get("panelsJSON", "[]"))
links = sum(1 for line in landing_markdown(hosts).splitlines() if line.startswith("| **["))
print(f"dashboard: {stored.get('id')} | title {stored.get('attributes', {}).get('title')!r} | "
f"{len(panels)} panel(s) stored | {links} links in the markdown")
if not panels:
raise SystemExit(" the dashboard saved but with no panels — Kibana rejected the panel shape")
print("open:", f"/app/dashboards#/view/{DASHBOARD_ID}")
route = f"/app/dashboards#/view/{DASHBOARD_ID}"
req("POST", "/api/kibana/settings", {"changes": {"defaultRoute": route}})
now = req("GET", "/api/kibana/settings").get("settings", {}).get("defaultRoute", {}).get("userValue")
print("defaultRoute:", now or "(not set!)")
if now != route:
raise SystemExit(" Kibana did not keep that defaultRoute — set it by hand in Advanced Settings")