Triage - bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up. - The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate read, and allows retry/bin only for the admin groups; the path validation is the single thing between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside the streams tree. - The sweep logs event=parked once per run so Kibana can name stuck files rather than only count them. Landing - diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management. Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats' setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed. - /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and verifies each step by reading it back. Installers - bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh, configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which nothing did before. - check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node binary that is not there is a skip, not a failure. Fixes - Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned. - HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back root-relative and landed on the SPA catch-all. header_down rewrites Location. - Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately. - Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit. - Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
112 lines
5.9 KiB
Bash
Executable File
112 lines
5.9 KiB
Bash
Executable File
#!/bin/bash
|
|
# Debian 13 host installer for vanguard-stream. Run with sudo from this directory.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")"
|
|
[[ $EUID -eq 0 ]] || { echo "run with sudo"; exit 1; }
|
|
OWNER="${SUDO_USER:-debian}" # the login user; owns /srv/media and may run docker
|
|
|
|
echo "== packages"
|
|
apt-get update -q
|
|
apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 python3-yaml xfsprogs git nodejs npm rsync
|
|
|
|
if ! command -v docker >/dev/null; then
|
|
echo "== docker"
|
|
install -m 0755 -d /etc/apt/keyrings
|
|
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
|
|
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
|
|
> /etc/apt/sources.list.d/docker.list
|
|
apt-get update -q
|
|
apt-get install -y -q docker-ce docker-ce-cli containerd.io docker-compose-plugin
|
|
fi
|
|
|
|
echo "== timezone: server clock, logs, timers and the dump window all in Eastern"
|
|
timedatectl set-timezone "${VG_TZ:-America/New_York}"
|
|
timedatectl set-ntp true
|
|
echo " timezone is: $(timedatectl show -p Timezone --value) (override: VG_TZ=... bash install.sh)"
|
|
|
|
echo "== docker waits for /srv/media (never record onto the OS disk)"
|
|
install -d /etc/systemd/system/docker.service.d
|
|
cat > /etc/systemd/system/docker.service.d/10-srv-media.conf <<'DROP'
|
|
[Unit]
|
|
RequiresMountsFor=/srv/media
|
|
DROP
|
|
systemctl daemon-reload
|
|
systemctl restart docker 2>/dev/null || true
|
|
|
|
echo "== media user + folders"
|
|
# media files are owned by uid 1000 — on OVH's template that is the 'debian' user
|
|
getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
|
|
usermod -aG docker "$OWNER" 2>/dev/null || true
|
|
install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
|
|
install -d -o 1000 -g 1000 /srv/media/.markers/metrics # script counters -> dumpd /metrics
|
|
install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
|
|
install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
|
|
install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
|
|
install -d -o 1000 -g 1000 /srv/bot /srv/bot/app # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
|
|
install -d -o 1000 -g 1000 /srv/appdata/elasticsearch # the ES image runs as uid 1000
|
|
install -d /var/lib/vanguard # deadman state
|
|
# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
|
|
sysctl -q -w vm.max_map_count=262144
|
|
printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
|
|
install -d -o "$OWNER" /srv/live # live portal static build (deployLive.sh)
|
|
install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
|
|
for d in /srv/media/clips/*/; do
|
|
[[ -e "$d/.stignore" ]] || install -m 0644 -o 1000 -g 1000 syncthing/stignore "$d/.stignore"
|
|
[[ -e "$d/.stlocal" ]] || install -m 0644 -o 1000 -g 1000 /dev/null "$d/.stlocal" # #include'd by .stignore
|
|
done
|
|
|
|
echo "== host scripts + units"
|
|
# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept
|
|
# copies had drifted, so a push-to-deploy was installing half of them.
|
|
grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do
|
|
install -m "$mode" "$path" /usr/local/bin/
|
|
done
|
|
install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot
|
|
install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it
|
|
install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord
|
|
# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
|
|
# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
|
|
if [[ ! -f /etc/vanguard/alert-webhook ]]; then
|
|
install -d -m 0750 /etc/vanguard
|
|
install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook # root timers are the only readers
|
|
fi
|
|
install -d -m 0750 /etc/vanguard
|
|
install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
|
|
install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
|
|
systemctl daemon-reload
|
|
systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
|
|
|
|
echo "== sysctl"
|
|
cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'
|
|
vm.swappiness = 10
|
|
net.core.rmem_max = 16777216
|
|
net.core.wmem_max = 16777216
|
|
SYS
|
|
sysctl -q --system
|
|
|
|
echo "== compose images (the stack is started by ./configure.py --full, which generates .env first)"
|
|
chown -R "$OWNER:" .
|
|
docker compose pull -q --ignore-buildable 2>/dev/null || docker compose pull -q || true
|
|
|
|
echo "== live portal: build into /srv/live (node)"
|
|
if [[ -d live && -f live/package.json ]]; then
|
|
( cd live && sudo -u "$OWNER" -H npm install --no-audit --no-fund --loglevel=error && sudo -u "$OWNER" -H npm run build ) \
|
|
&& install -d -o "$OWNER" /srv/live && rsync -a --delete live/dist/ /srv/live/ && echo " built" || echo " portal build failed (run ./deployLive.sh from a dev box later)"
|
|
fi
|
|
|
|
echo "== host firewall (nftables; own table, input only; keeps this SSH session alive)"
|
|
if [[ -f firewall/firewall.sh ]]; then
|
|
bash firewall/firewall.sh
|
|
else
|
|
echo " firewall/firewall.sh missing — skipped"
|
|
fi
|
|
|
|
cat <<'MSG'
|
|
|
|
== done. Next: ./configure.py --full (hostnames, passwords, .env secrets; starts the containers)
|
|
== firewall is active: open a SECOND terminal and confirm ssh still works before closing this one.
|
|
|
|
== log out and back in once so the docker group applies; then 'docker compose' works without sudo.
|
|
== syncthing UI: ssh -L 8384:127.0.0.1:8384 debian@SERVER -> http://localhost:8384
|
|
MSG
|