Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
109 lines
5.8 KiB
Bash
Executable File
109 lines
5.8 KiB
Bash
Executable File
#!/bin/bash
|
|
# Debian 13 host installer for vanguard-stream. Run with sudo from this directory.
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")"
|
|
[[ $EUID -eq 0 ]] || { echo "run with sudo"; exit 1; }
|
|
OWNER="${SUDO_USER:-debian}" # the login user; owns /srv/media and may run docker
|
|
|
|
echo "== packages"
|
|
apt-get update -q
|
|
apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 python3-yaml xfsprogs git nodejs npm rsync
|
|
|
|
if ! command -v docker >/dev/null; then
|
|
echo "== docker"
|
|
install -m 0755 -d /etc/apt/keyrings
|
|
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
|
|
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
|
|
> /etc/apt/sources.list.d/docker.list
|
|
apt-get update -q
|
|
apt-get install -y -q docker-ce docker-ce-cli containerd.io docker-compose-plugin
|
|
fi
|
|
|
|
echo "== timezone: server clock, logs, timers and the dump window all in Eastern"
|
|
timedatectl set-timezone "${VG_TZ:-America/New_York}"
|
|
timedatectl set-ntp true
|
|
echo " timezone is: $(timedatectl show -p Timezone --value) (override: VG_TZ=... bash install.sh)"
|
|
|
|
echo "== docker waits for /srv/media (never record onto the OS disk)"
|
|
install -d /etc/systemd/system/docker.service.d
|
|
cat > /etc/systemd/system/docker.service.d/10-srv-media.conf <<'DROP'
|
|
[Unit]
|
|
RequiresMountsFor=/srv/media
|
|
DROP
|
|
systemctl daemon-reload
|
|
systemctl restart docker 2>/dev/null || true
|
|
|
|
echo "== media user + folders"
|
|
# media files are owned by uid 1000 — on OVH's template that is the 'debian' user
|
|
getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
|
|
usermod -aG docker "$OWNER" 2>/dev/null || true
|
|
install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
|
|
install -d -o 1000 -g 1000 /srv/media/.markers/metrics # script counters -> dumpd /metrics
|
|
install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
|
|
install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
|
|
install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
|
|
install -d -o 1000 -g 1000 /srv/bot /srv/bot/app # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
|
|
install -d -o 1000 -g 1000 /srv/appdata/elasticsearch # the ES image runs as uid 1000
|
|
install -d /var/lib/vanguard # deadman state
|
|
# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
|
|
sysctl -q -w vm.max_map_count=262144
|
|
printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
|
|
install -d -o "$OWNER" /srv/live # live portal static build (deployLive.sh)
|
|
install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
|
|
for d in /srv/media/clips/*/; do
|
|
[[ -e "$d/.stignore" ]] || install -m 0644 -o 1000 -g 1000 syncthing/stignore "$d/.stignore"
|
|
[[ -e "$d/.stlocal" ]] || install -m 0644 -o 1000 -g 1000 /dev/null "$d/.stlocal" # #include'd by .stignore
|
|
done
|
|
|
|
echo "== host scripts + units"
|
|
install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
|
|
install -m 0644 bin/vgmetrics.sh /usr/local/bin/ # sourced by the sweep/vgdeband; counters for dumpd's /metrics
|
|
install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot
|
|
install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it
|
|
install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord
|
|
# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
|
|
# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
|
|
if [[ ! -f /etc/vanguard/alert-webhook ]]; then
|
|
install -d -m 0750 /etc/vanguard
|
|
install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook # root timers are the only readers
|
|
fi
|
|
install -d -m 0750 /etc/vanguard
|
|
install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
|
|
install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
|
|
systemctl daemon-reload
|
|
systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
|
|
|
|
echo "== sysctl"
|
|
cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'
|
|
vm.swappiness = 10
|
|
net.core.rmem_max = 16777216
|
|
net.core.wmem_max = 16777216
|
|
SYS
|
|
sysctl -q --system
|
|
|
|
echo "== compose images (the stack is started by ./configure.py --full, which generates .env first)"
|
|
chown -R "$OWNER:" .
|
|
docker compose pull -q --ignore-buildable 2>/dev/null || docker compose pull -q || true
|
|
|
|
echo "== live portal: build into /srv/live (node)"
|
|
if [[ -d live && -f live/package.json ]]; then
|
|
( cd live && sudo -u "$OWNER" -H npm install --no-audit --no-fund --loglevel=error && sudo -u "$OWNER" -H npm run build ) \
|
|
&& install -d -o "$OWNER" /srv/live && rsync -a --delete live/dist/ /srv/live/ && echo " built" || echo " portal build failed (run ./deployLive.sh from a dev box later)"
|
|
fi
|
|
|
|
echo "== host firewall (nftables; own table, input only; keeps this SSH session alive)"
|
|
if [[ -f firewall/firewall.sh ]]; then
|
|
bash firewall/firewall.sh
|
|
else
|
|
echo " firewall/firewall.sh missing — skipped"
|
|
fi
|
|
|
|
cat <<'MSG'
|
|
|
|
== done. Next: ./configure.py --full (hostnames, passwords, .env secrets; starts the containers)
|
|
== firewall is active: open a SECOND terminal and confirm ssh still works before closing this one.
|
|
|
|
== log out and back in once so the docker group applies; then 'docker compose' works without sudo.
|
|
== syncthing UI: ssh -L 8384:127.0.0.1:8384 debian@SERVER -> http://localhost:8384
|
|
MSG
|