diff --git a/CommunityDiscordMedia.md b/CommunityDiscordMedia.md index ff1b074..320d44c 100644 --- a/CommunityDiscordMedia.md +++ b/CommunityDiscordMedia.md @@ -320,51 +320,39 @@ permanent reference is needed for long sessions, not just a start burst. export at 2560×1440) - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode) -### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (NEXT) +### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (built; needs DNS + a webhook) -**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`. -Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one -host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer). -VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already -expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector. +**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a +growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two +services and a dashboard tool we would only use as a log viewer). The framework everything feeds is +`LOGGING.md`; as-built detail lives there and in README §5h. +Six containers, all loopback, published only through the gated `diag.` host: +`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`. + +- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL` + asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534) +- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose + bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start), + the deadman script and timer +- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so + a critical disk does not also report as filling +- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat + through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery + once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal +- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in + `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy - [ ] CNAME `diag` → the main host -- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`) -- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with - `-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not** - strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans. -- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep, - trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The - `vanguard.service` / `vanguard.category` labels already on every container become stream fields, so - filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them - (JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx, - multiline + regex for jellyfin). -- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API, - and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The - alternative is a read-only socket proxy in front of it, one more container but no root. Reading the - json files directly avoids both and loses the container labels, which is most of the value. -- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy` - longer than the rest would mean separate tenants — not worth it unless an audit needs it. -- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live - (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs - would consume them. Not required for logs. -- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query` - (+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against - those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain - covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy. - Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics - scraping the four taps that are already live. Full picture: four containers. -- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv` - over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error` - in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the - failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up - version for thresholds, grouping and silencing, not a prerequisite for being warned. - -**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an -admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access. -Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now, -purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a -logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command". +- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env` +- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run) +- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing, + `journalctl -u vanguard-deadman` is quiet +- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be + checked without the services running +- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong + way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every + invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web + page with Docker access, never a generic command runner. ### Phase 14 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/) - server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved diff --git a/LOGGING.md b/LOGGING.md index 3f7eaa9..a189ea9 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -84,7 +84,30 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping `auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does. -## Wiring sketches (not deployed; pick one later) +## As built (Phase 13) + +| container | port | job | +|---|---|---| +| `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged | +| `vector` | — | `docker_logs` + `journald` → VictoriaLogs (elasticsearch bulk). **Holds the Docker socket** | +| `victoriametrics` | 8428 | 90 d, scrapes the taps above (`observability/scrape.yml`) | +| `vmalert-metrics` | 8880 | `observability/alerts-metrics.yml` against VictoriaMetrics | +| `vmalert-logs` | 8881 | `observability/alerts-logs.yml` (LogsQL) against VictoriaLogs' stats endpoints | +| `alertmanager` | 9093 | groups, inhibits, posts to Discord (`webhook_url_file`, never in the config) | + +Published only through `diag.BASE`, gated on `DIAG_REQUIRE_GROUP` (`media-admins`): `/logs`, `/metrics-db`, +`/alerts`, `/rules-metrics`, `/rules-logs`, and an index at `/`. + +**The deadman** (`bin/vanguard-deadman.sh`, host timer, 5 min) is deliberately outside all of it: curl only, it +checks each component answers, that the always-firing `Watchdog` alert is still arriving through +vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. It speaks on change, announces recovery once, and +with no webhook configured still logs and exits non-zero so `systemctl is-failed vanguard-deadman.service` is +a signal. Nothing inside the stack can report its own death; this can. + +`ALERT_WEBHOOK_URL` in `.env` → `/etc/vanguard/alert-webhook` (0640 root:65534). Empty means log-only, which +is a working configuration, not an error. + +## Wiring sketches (the options that were weighed) - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea). diff --git a/README.md b/README.md index 99d92c9..44c89f9 100644 --- a/README.md +++ b/README.md @@ -424,6 +424,37 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe Jellyfin and Gitea use OIDC providers instead, not this. +## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`) + +Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`, +`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands. + + https://diag.HOST/logs logs (VictoriaLogs; 30 days) + https://diag.HOST/metrics-db metrics (VictoriaMetrics; 90 days) + https://diag.HOST/alerts what is firing (Alertmanager) + https://diag.HOST/rules-metrics metric rules · /rules-logs log rules + +Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not +alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart +loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures, +publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook, +`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file. + +**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with +nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole +vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot +report its own death, and silence looks exactly like health. It reports on change only, announces recovery +once, and without a webhook still logs and exits non-zero so the unit's state is the signal. + + systemctl list-timers vanguard-deadman.timer + journalctl -u vanguard-deadman -n 20 + sudo /usr/local/bin/vanguard-deadman.sh # run it now + +Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker +socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen +deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the +journald log driver on every container or a read-only socket proxy in front of vector. + ## 6. Editors Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik diff --git a/STATUS.md b/STATUS.md index 73b020a..6565465 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,7 +1,7 @@ # Vanguard stack — what runs, what people do, what's unfinished Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01. -Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box. +Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box. Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`). ## Runs on its own @@ -59,7 +59,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | normalize sweep | editing standard, one copy per clip | live | | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live | | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) | -| Metrics taps | node_exporter (host), cAdvisor (containers), MediaMTX and Caddy metrics — Prometheus format on loopback, no store yet | **committed 2026-11-01, deploy pending** | +| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **committed 2026-11-03, deploy pending**: needs the `diag` CNAME and `ALERT_WEBHOOK_URL`; expect one round of corrections to Vector's journald fields and the LogsQL rules | +| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** | | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) | | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live | | Jellyfin | watch page, SSO via Authentik, direct play only | live | diff --git a/bin/vanguard-deadman.sh b/bin/vanguard-deadman.sh index 1e56b93..82dff95 100755 --- a/bin/vanguard-deadman.sh +++ b/bin/vanguard-deadman.sh @@ -14,6 +14,10 @@ # # State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes, # and says so once when everything recovers. +# +# No webhook configured? The checks still run and still print to the journal, and the unit exits non-zero +# when it found something it could not report — so `systemctl is-failed vanguard-deadman.service` and +# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone. set -uo pipefail WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}" @@ -57,20 +61,40 @@ was="$(cat "$STATE" 2>/dev/null || true)" printf '%s' "$now" > "$STATE" [[ $now == "$was" ]] && exit 0 # nothing changed since the last run: stay quiet -post() { # post - [[ -s $WEBHOOK_FILE ]] || { echo "deadman: no webhook in $WEBHOOK_FILE; not sent: $1" >&2; return; } +# The webhook is optional. With none configured the checks still run and still land in the journal +# (journalctl -u vanguard-deadman), so nothing silently stops working; only the Discord delivery is skipped. +webhook="" +if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi +case "$webhook" in + https://*) ;; + "") webhook="" ;; + *) echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;; +esac + +post() { # post ; a no-op without a webhook + [[ -n $webhook ]] || return 1 curl -s -m 15 -H 'Content-Type: application/json' \ --data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \ - "$(cat "$WEBHOOK_FILE")" >/dev/null || echo "deadman: webhook post failed" >&2 + "$webhook" >/dev/null && return 0 + echo "deadman: the webhook post failed" >&2 + return 1 } if [[ -z $now ]]; then - echo "deadman: recovered" - post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again." -else - printf 'deadman: %s\n' "${problems[@]}" - body="⚠️ **Vanguard monitoring is not healthy** (this message comes from the host, outside the stack):" - for item in "${problems[@]}"; do body+=$'\n'"• $item"; done - body+=$'\n'"Until this clears, assume no other alert can reach you." - post "$body" + echo "deadman: recovered — the alerting stack and disks are healthy again" + post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again." || true + exit 0 fi + +printf 'deadman: %s\n' "${problems[@]}" >&2 +body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside the stack):" +for item in "${problems[@]}"; do body+=$'\n'"• $item"; done +body+=$'\n'"Until this clears, assume no other alert can reach you." +if post "$body"; then exit 0; fi + +# Problems found and nobody could be told. Exit non-zero so the unit shows as failed: with no webhook +# configured, `systemctl is-failed vanguard-deadman.service` is the only signal there is. +if [[ -z $webhook ]]; then + echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2 +fi +exit 1 diff --git a/check.sh b/check.sh index 9ef5f92..5a37aea 100755 --- a/check.sh +++ b/check.sh @@ -13,7 +13,7 @@ for f in configure.py bin/dumpd.py bin/vgstrip.py; do python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; } done rm -rf __pycache__ bin/__pycache__ -python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } +python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/vector.yaml observability/scrape.yml observability/alerts-metrics.yml observability/alerts-logs.yml observability/alertmanager.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } node -e " const fs=require('fs'); for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } diff --git a/configure.py b/configure.py index e64346e..9dca7cd 100755 --- a/configure.py +++ b/configure.py @@ -580,7 +580,9 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis e = read_env() Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600) Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600) - run(["sudo", "install", "-m", "0644", "alert-webhook", "/etc/vanguard/alert-webhook"]) # read by alertmanager (uid 65534) and the deadman + # 0640 root:65534 — the alertmanager image runs as nobody(65534) and the deadman runs as root. A webhook + # URL is a credential for that channel, so it is not world-readable. + run(["sudo", "install", "-m", "0640", "-o", "root", "-g", "65534", "alert-webhook", "/etc/vanguard/alert-webhook"]) run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")): run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"]) @@ -652,6 +654,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_ _e = read_env() if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}") print(f"== live : https://{live_host}/ (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live : (not set — run --full)") + alert_hook = read_env().get("ALERT_WEBHOOK_URL", "") + print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); deadman posts there too" if alert_hook + else "NO WEBHOOK — alerts and the deadman only write to the journal. Set ALERT_WEBHOOK_URL in .env.")) print(f"== diag : https://{diag_host}/logs (logs, metrics, alerts; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)") print(f"== bot web : https://{bot_host}/ (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web : (not set — run --full)") print(f"== gitea : https://{git_host}/ ssh port 2222 admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')} (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea : (not set — run --full)") @@ -713,6 +718,9 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_ print("\n-- Discord bot (optional; the Bot added to the Authentik Discord application). Enter keeps the current value.") discord_token, discord_guild = ask_discord() live_webhook = ask(" Discord webhook URL for 'is live' notices (optional)", read_env().get("LIVE_WEBHOOK_URL", "")) + env0 = read_env() + alert_webhook = ask("Discord webhook for ALERTS and the deadman (blank = log only, no Discord)", + env0.get("ALERT_WEBHOOK_URL", "")) print("\n-- streamers (OBS publishers; each gets streams/ and clips/)") for u in list(streamers): k = ask(f" {u}: [k]eep [r]emove [p]assword reset", "k").lower()[:1] @@ -736,6 +744,9 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_ Path(".env").write_text("".join(f"{k}={v}\n" for k, v in e.items())) if Path(".env").exists() else None if not Path(".env").exists(): Path(".env").write_text(f"APEX_HOST={apex}\nAPEX_REDIRECT={apex_to}\n"); os.chmod(".env", 0o600) + if alert_webhook != env0.get("ALERT_WEBHOOK_URL", ""): + env = read_env(); env["ALERT_WEBHOOK_URL"] = alert_webhook + Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items())); os.chmod(".env", 0o600) return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host def ask_discord(): diff --git a/install.sh b/install.sh index faebe3a..8ca33d8 100755 --- a/install.sh +++ b/install.sh @@ -42,6 +42,9 @@ install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/aut install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner install -d -o 1000 -g 1000 /srv/bot /srv/bot/app # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md) +install -d -o 1000 -g 1000 /srv/appdata/victorialogs /srv/appdata/victoriametrics # run as uid 1000 +install -d -o 65534 -g 65534 /srv/appdata/alertmanager # prom/alertmanager runs as nobody +install -d /var/lib/vanguard # deadman state install -d -o "$OWNER" /srv/live # live portal static build (deployLive.sh) install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire} for d in /srv/media/clips/*/; do @@ -52,11 +55,18 @@ done echo "== host scripts + units" install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/ install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot +install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it +# Compose bind-mounts this as a FILE; without it Docker would create a directory and alertmanager would not +# start. configure.py rewrites it from ALERT_WEBHOOK_URL on every apply; an empty one means "log only". +if [[ ! -f /etc/vanguard/alert-webhook ]]; then + install -d -m 0750 /etc/vanguard + install -m 0640 -o root -g 65534 /dev/null /etc/vanguard/alert-webhook +fi install -d -m 0750 /etc/vanguard install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/ systemctl daemon-reload -systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service +systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer echo "== sysctl" cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'