Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
read, and allows retry/bin only for the admin groups; the path validation is the single thing
between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
them.
Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
verifies each step by reading it back.
Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
binary that is not there is a skip, not a failure.
Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.
Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
with zero harvesters and no index. Container logs now use filestream + the container parser
with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.
Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
the user holds a privilege and that API does not exist when security is off (the
Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
the only gate. configure.py generates the service passwords, creates the accounts and writes
a root-only netrc for the deadman; --es-security retries just that step.
Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
that the stack's own alerting died.
Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
(loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.
Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for
Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers
host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and
Caddy's Prometheus endpoints.
Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box
whose spare capacity belongs to x264), single-node discovery, security off because it only listens on
loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start
without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart.
diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links.
The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is
not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when
space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually
written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the
disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each service now runs behind the prefix it is published under (-http.pathPrefix, or route-prefix plus
external-url for Alertmanager). Without that, three of the four UIs would have loaded at their sub-path and
then fetched their assets from the domain root, so the pages would have come up broken. The prefixes cascade,
so the scrape paths, vmalert's notifier and remote-write URLs, and the deadman's health probes all follow.
The index is diag/index.html mounted into Caddy, not an inline respond: Caddy reads {...} as placeholders and
the page's CSS is full of braces, so inline would have been mangled. It also matches how the gateway, dock and
clips pages are already served.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six containers, all loopback-published, reachable only through the gated diag. host:
victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
victoriametrics 90d, scrapes the four taps that were already live plus the new services
vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config
The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.
Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
authentik/vanguard-profile-scope-mapping.py (new, and the first time that mapping is version-controlled rather
than pasted from the docs) emits the Discord snowflake from the account's source connection alongside the
inherited groups, so the bot never has to match a name.
docker-compose.yml: discord-bot moves to its own network. It trusts the identity headers Caddy injects, and it
shared the default bridge with jellyfin, copyparty, syncthing, gitea, act_runner and cadvisor, any of which
could reach :8000 directly and forge them. It needs no sibling container: Caddy uses the published loopback
port and MariaDB is reached through host.docker.internal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row,
README and the plan doc all describe the installed-release model instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image no longer installs the bot's dependencies: its CI builds one tar.gz holding the app plus a
vendor/ directory of prebuilt wheels, and PYTHONPATH=/app/vendor makes the code import them. A new
dependency therefore needs no image rebuild and the server contacts no package index.
- deployBotBundle.sh: install / --latest / --status / --rollback, verifies every file against BUILD.json
and keeps the previous release beside the live one.
- compose: mount ${BOT_APP_DIR:-/srv/bot/app} at /app, set DB_BACKEND, DATA_DIR, WEB_ALLOWED_GROUPS,
WEB_SERVE_UI and PYTHONPATH. bot/app placeholder untracked; install.sh creates /srv/bot/app.
- The placeholder requirements would have crashed the bot on import (no sqlglot), which is why the
image now ships none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bin/mtx-on-ready.sh (runOnReady inside the mediamtx container) posts the is-live notice and, for sources whose
audio is not Opus, runs ffmpeg: video copy, audio -> Opus, published over MediaMTX's loopback RTSP to
<user>/<Title>.webrtc (record: no). dumpd accepts .webrtc publishes only from 127.0.0.1. The portal hides
mirror paths, plays the mirror over WHEP (HLS stays on the source), holds an AAC source for 15 s while its
mirror comes up, and corrects the viewer count. Verified headless with a mocked path list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
