Files
Media-Server-Setup/install.sh
Brandyn / Techy ca01516d42 Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms
Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 21:25:43 -06:00

112 lines
5.9 KiB
Bash
Executable File

#!/bin/bash
# Debian 13 host installer for vanguard-stream. Run with sudo from this directory.
set -euo pipefail
cd "$(dirname "$0")"
[[ $EUID -eq 0 ]] || { echo "run with sudo"; exit 1; }
OWNER="${SUDO_USER:-debian}" # the login user; owns /srv/media and may run docker
echo "== packages"
apt-get update -q
apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 python3-yaml xfsprogs git nodejs npm rsync
if ! command -v docker >/dev/null; then
echo "== docker"
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -q
apt-get install -y -q docker-ce docker-ce-cli containerd.io docker-compose-plugin
fi
echo "== timezone: server clock, logs, timers and the dump window all in Eastern"
timedatectl set-timezone "${VG_TZ:-America/New_York}"
timedatectl set-ntp true
echo " timezone is: $(timedatectl show -p Timezone --value) (override: VG_TZ=... bash install.sh)"
echo "== docker waits for /srv/media (never record onto the OS disk)"
install -d /etc/systemd/system/docker.service.d
cat > /etc/systemd/system/docker.service.d/10-srv-media.conf <<'DROP'
[Unit]
RequiresMountsFor=/srv/media
DROP
systemctl daemon-reload
systemctl restart docker 2>/dev/null || true
echo "== media user + folders"
# media files are owned by uid 1000 — on OVH's template that is the 'debian' user
getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
usermod -aG docker "$OWNER" 2>/dev/null || true
install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
install -d -o 1000 -g 1000 /srv/media/.markers/metrics # script counters -> dumpd /metrics
install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
install -d -o 1000 -g 1000 /srv/bot /srv/bot/app # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
install -d -o 1000 -g 1000 /srv/appdata/elasticsearch # the ES image runs as uid 1000
install -d /var/lib/vanguard # deadman state
# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
sysctl -q -w vm.max_map_count=262144
printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
install -d -o "$OWNER" /srv/live # live portal static build (deployLive.sh)
install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
for d in /srv/media/clips/*/; do
[[ -e "$d/.stignore" ]] || install -m 0644 -o 1000 -g 1000 syncthing/stignore "$d/.stignore"
[[ -e "$d/.stlocal" ]] || install -m 0644 -o 1000 -g 1000 /dev/null "$d/.stlocal" # #include'd by .stignore
done
echo "== host scripts + units"
# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept
# copies had drifted, so a push-to-deploy was installing half of them.
grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do
install -m "$mode" "$path" /usr/local/bin/
done
install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot
install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it
install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord
# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
if [[ ! -f /etc/vanguard/alert-webhook ]]; then
install -d -m 0750 /etc/vanguard
install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook # root timers are the only readers
fi
install -d -m 0750 /etc/vanguard
install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
echo "== sysctl"
cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'
vm.swappiness = 10
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
SYS
sysctl -q --system
echo "== compose images (the stack is started by ./configure.py --full, which generates .env first)"
chown -R "$OWNER:" .
docker compose pull -q --ignore-buildable 2>/dev/null || docker compose pull -q || true
echo "== live portal: build into /srv/live (node)"
if [[ -d live && -f live/package.json ]]; then
( cd live && sudo -u "$OWNER" -H npm install --no-audit --no-fund --loglevel=error && sudo -u "$OWNER" -H npm run build ) \
&& install -d -o "$OWNER" /srv/live && rsync -a --delete live/dist/ /srv/live/ && echo " built" || echo " portal build failed (run ./deployLive.sh from a dev box later)"
fi
echo "== host firewall (nftables; own table, input only; keeps this SSH session alive)"
if [[ -f firewall/firewall.sh ]]; then
bash firewall/firewall.sh
else
echo " firewall/firewall.sh missing — skipped"
fi
cat <<'MSG'
== done. Next: ./configure.py --full (hostnames, passwords, .env secrets; starts the containers)
== firewall is active: open a SECOND terminal and confirm ssh still works before closing this one.
== log out and back in once so the docker group applies; then 'docker compose' works without sudo.
== syncthing UI: ssh -L 8384:127.0.0.1:8384 debian@SERVER -> http://localhost:8384
MSG