Phase 13 integration: install.sh, configure.py, and a deadman that degrades cleanly

install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.

configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.

The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.

check.sh validates all five observability configs; confirmed it catches a deliberate break.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -320,51 +320,39 @@ permanent reference is needed for long sessions, not just a start burst.
       export at 2560×1440)
 - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
 
-### Phase 13 — `diag.` host: logs, metrics, alerting, deadman  (NEXT)
+### Phase 13 — `diag.` host: logs, metrics, alerting, deadman  (built; needs DNS + a webhook)
 
-**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
+**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a
-Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
+growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two
-host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
+services and a dashboard tool we would only use as a log viewer). The framework everything feeds is
-VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
+`LOGGING.md`; as-built detail lives there and in README §5h.
-expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
 
+Six containers, all loopback, published only through the gated `diag.` host:
+`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`.
+
+- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL`
+      asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534)
+- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose
+      bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start),
+      the deadman script and timer
+- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so
+      a critical disk does not also report as filling
+- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat
+      through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery
+      once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal
+- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in
+      `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
 - [ ] CNAME `diag` → the main host
-- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
+- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
-- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
+- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run)
-      `-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
+- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
-      strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
+      `journalctl -u vanguard-deadman` is quiet
-- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
+- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be
-      trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
+      checked without the services running
-      `vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
+- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
-      filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
+      way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
-      (JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
+      invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
-      multiline + regex for jellyfin).
+      page with Docker access, never a generic command runner.
-- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
-      and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
-      alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
-      json files directly avoids both and loses the container labels, which is most of the value.
-- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
-      longer than the rest would mean separate tenants — not worth it unless an audit needs it.
-- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
-      (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
-      would consume them. Not required for logs.
-- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
-      (+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
-      those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
-      covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
-      Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
-      scraping the four taps that are already live. Full picture: four containers.
-- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
-      over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
-      in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
-      failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
-      version for thresholds, grouping and silencing, not a prerequisite for being warned.
-
-**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
-admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
-Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
-purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
-logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
 
 ### Phase 14 — push-to-deploy  (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
 - server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved

LOGGING.md

@@ -84,7 +84,30 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
 One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
 `auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
 
-## Wiring sketches (not deployed; pick one later)
+## As built (Phase 13)
+
+| container | port | job |
+|---|---|---|
+| `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged |
+| `vector` | — | `docker_logs` + `journald` → VictoriaLogs (elasticsearch bulk). **Holds the Docker socket** |
+| `victoriametrics` | 8428 | 90 d, scrapes the taps above (`observability/scrape.yml`) |
+| `vmalert-metrics` | 8880 | `observability/alerts-metrics.yml` against VictoriaMetrics |
+| `vmalert-logs` | 8881 | `observability/alerts-logs.yml` (LogsQL) against VictoriaLogs' stats endpoints |
+| `alertmanager` | 9093 | groups, inhibits, posts to Discord (`webhook_url_file`, never in the config) |
+
+Published only through `diag.BASE`, gated on `DIAG_REQUIRE_GROUP` (`media-admins`): `/logs`, `/metrics-db`,
+`/alerts`, `/rules-metrics`, `/rules-logs`, and an index at `/`.
+
+**The deadman** (`bin/vanguard-deadman.sh`, host timer, 5 min) is deliberately outside all of it: curl only, it
+checks each component answers, that the always-firing `Watchdog` alert is still arriving through
+vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. It speaks on change, announces recovery once, and
+with no webhook configured still logs and exits non-zero so `systemctl is-failed vanguard-deadman.service` is
+a signal. Nothing inside the stack can report its own death; this can.
+
+`ALERT_WEBHOOK_URL` in `.env` → `/etc/vanguard/alert-webhook` (0640 root:65534). Empty means log-only, which
+is a working configuration, not an error.
+
+## Wiring sketches (the options that were weighed)
 
 - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
   labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).

README.md

@@ -424,6 +424,37 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
 
 Jellyfin and Gitea use OIDC providers instead, not this.
 
+## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`)
+
+Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`,
+`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands.
+
+    https://diag.HOST/logs            logs (VictoriaLogs; 30 days)
+    https://diag.HOST/metrics-db      metrics (VictoriaMetrics; 90 days)
+    https://diag.HOST/alerts          what is firing (Alertmanager)
+    https://diag.HOST/rules-metrics   metric rules · /rules-logs  log rules
+
+Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not
+alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart
+loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures,
+publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook,
+`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file.
+
+**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
+nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole
+vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot
+report its own death, and silence looks exactly like health. It reports on change only, announces recovery
+once, and without a webhook still logs and exits non-zero so the unit's state is the signal.
+
+    systemctl list-timers vanguard-deadman.timer
+    journalctl -u vanguard-deadman -n 20
+    sudo /usr/local/bin/vanguard-deadman.sh          # run it now
+
+Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker
+socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen
+deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the
+journald log driver on every container or a read-only socket proxy in front of vector.
+
 ## 6. Editors
 
 Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik

STATUS.md

@@ -1,7 +1,7 @@
 # Vanguard stack — what runs, what people do, what's unfinished
 
 Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01.
-Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
+Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
 Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).
 
 ## Runs on its own
@@ -59,7 +59,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
-| Metrics taps | node_exporter (host), cAdvisor (containers), MediaMTX and Caddy metrics — Prometheus format on loopback, no store yet | **committed 2026-11-01, deploy pending** |
+| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **committed 2026-11-03, deploy pending**: needs the `diag` CNAME and `ALERT_WEBHOOK_URL`; expect one round of corrections to Vector's journald fields and the LogsQL rules |
+| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
 | Jellyfin | watch page, SSO via Authentik, direct play only | live |

bin/vanguard-deadman.sh

@@ -14,6 +14,10 @@
 #
 # State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes,
 # and says so once when everything recovers.
+#
+# No webhook configured? The checks still run and still print to the journal, and the unit exits non-zero
+# when it found something it could not report — so `systemctl is-failed vanguard-deadman.service` and
+# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
 set -uo pipefail
 
 WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
@@ -57,20 +61,40 @@ was="$(cat "$STATE" 2>/dev/null || true)"
 printf '%s' "$now" > "$STATE"
 [[ $now == "$was" ]] && exit 0                      # nothing changed since the last run: stay quiet
 
-post() {                                            # post <text>
+# The webhook is optional. With none configured the checks still run and still land in the journal
-  [[ -s $WEBHOOK_FILE ]] || { echo "deadman: no webhook in $WEBHOOK_FILE; not sent: $1" >&2; return; }
+# (journalctl -u vanguard-deadman), so nothing silently stops working; only the Discord delivery is skipped.
+webhook=""
+if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
+case "$webhook" in
+  https://*) ;;
+  "")        webhook="" ;;
+  *)         echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
+esac
+
+post() {                                            # post <text>; a no-op without a webhook
+  [[ -n $webhook ]] || return 1
   curl -s -m 15 -H 'Content-Type: application/json' \
     --data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
-    "$(cat "$WEBHOOK_FILE")" >/dev/null || echo "deadman: webhook post failed" >&2
+    "$webhook" >/dev/null && return 0
+  echo "deadman: the webhook post failed" >&2
+  return 1
 }
 
 if [[ -z $now ]]; then
-  echo "deadman: recovered"
+  echo "deadman: recovered — the alerting stack and disks are healthy again"
-  post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again."
+  post "✅ **Vanguard monitoring recovered** — the alerting stack and disks are healthy again." || true
-else
+  exit 0
-  printf 'deadman: %s\n' "${problems[@]}"
-  body="⚠️ **Vanguard monitoring is not healthy** (this message comes from the host, outside the stack):"
-  for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
-  body+=$'\n'"Until this clears, assume no other alert can reach you."
-  post "$body"
 fi
+
+printf 'deadman: %s\n' "${problems[@]}" >&2
+body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside the stack):"
+for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
+body+=$'\n'"Until this clears, assume no other alert can reach you."
+if post "$body"; then exit 0; fi
+
+# Problems found and nobody could be told. Exit non-zero so the unit shows as failed: with no webhook
+# configured, `systemctl is-failed vanguard-deadman.service` is the only signal there is.
+if [[ -z $webhook ]]; then
+  echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
+fi
+exit 1

check.sh

@@ -13,7 +13,7 @@ for f in configure.py bin/dumpd.py bin/vgstrip.py; do
   python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
 done
 rm -rf __pycache__ bin/__pycache__
-python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
+python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/vector.yaml observability/scrape.yml observability/alerts-metrics.yml observability/alerts-logs.yml observability/alertmanager.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
 node -e "
 const fs=require('fs');
 for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }

configure.py

@@ -580,7 +580,9 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     e = read_env()
     Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
     Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
-    run(["sudo", "install", "-m", "0644", "alert-webhook", "/etc/vanguard/alert-webhook"])   # read by alertmanager (uid 65534) and the deadman
+    # 0640 root:65534 — the alertmanager image runs as nobody(65534) and the deadman runs as root. A webhook
+    # URL is a credential for that channel, so it is not world-readable.
+    run(["sudo", "install", "-m", "0640", "-o", "root", "-g", "65534", "alert-webhook", "/etc/vanguard/alert-webhook"])
     run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"])           # read by dumpd (/deploy webhook) and deploy.sh
     if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
         run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
@@ -652,6 +654,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     _e = read_env()
     if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex     : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}")
     print(f"== live     : https://{live_host}/  (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live     : (not set — run --full)")
+    alert_hook = read_env().get("ALERT_WEBHOOK_URL", "")
+    print("== alerts   : " + (f"Discord webhook set ({alert_hook[:38]}…); deadman posts there too" if alert_hook
+          else "NO WEBHOOK — alerts and the deadman only write to the journal. Set ALERT_WEBHOOK_URL in .env."))
     print(f"== diag     : https://{diag_host}/logs  (logs, metrics, alerts; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag     : (not set — run --full)")
     print(f"== bot web  : https://{bot_host}/  (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web  : (not set — run --full)")
     print(f"== gitea    : https://{git_host}/  ssh port 2222  admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')}  (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea    : (not set — run --full)")
@@ -713,6 +718,9 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     print("\n-- Discord bot (optional; the Bot added to the Authentik Discord application). Enter keeps the current value.")
     discord_token, discord_guild = ask_discord()
     live_webhook = ask("  Discord webhook URL for 'is live' notices (optional)", read_env().get("LIVE_WEBHOOK_URL", ""))
+    env0 = read_env()
+    alert_webhook = ask("Discord webhook for ALERTS and the deadman (blank = log only, no Discord)",
+                        env0.get("ALERT_WEBHOOK_URL", ""))
     print("\n-- streamers (OBS publishers; each gets streams/<name> and clips/<name>)")
     for u in list(streamers):
         k = ask(f"  {u}: [k]eep  [r]emove  [p]assword reset", "k").lower()[:1]
@@ -736,6 +744,9 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     Path(".env").write_text("".join(f"{k}={v}\n" for k, v in e.items())) if Path(".env").exists() else None
     if not Path(".env").exists():
         Path(".env").write_text(f"APEX_HOST={apex}\nAPEX_REDIRECT={apex_to}\n"); os.chmod(".env", 0o600)
+    if alert_webhook != env0.get("ALERT_WEBHOOK_URL", ""):
+        env = read_env(); env["ALERT_WEBHOOK_URL"] = alert_webhook
+        Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items())); os.chmod(".env", 0o600)
     return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host
 
 def ask_discord():

install.sh

@@ -42,6 +42,9 @@ install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/aut
 install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
 install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
 install -d -o 1000 -g 1000 /srv/bot /srv/bot/app      # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
+install -d -o 1000 -g 1000 /srv/appdata/victorialogs /srv/appdata/victoriametrics   # run as uid 1000
+install -d -o 65534 -g 65534 /srv/appdata/alertmanager                             # prom/alertmanager runs as nobody
+install -d /var/lib/vanguard                                                       # deadman state
 install -d -o "$OWNER" /srv/live                 # live portal static build (deployLive.sh)
 install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
 for d in /srv/media/clips/*/; do
@@ -52,11 +55,18 @@ done
 echo "== host scripts + units"
 install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
 install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot   # dumpd runs this for the bot CI's POST /deploy-bot
+install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/      # watches the observability stack from outside it
+# Compose bind-mounts this as a FILE; without it Docker would create a directory and alertmanager would not
+# start. configure.py rewrites it from ALERT_WEBHOOK_URL on every apply; an empty one means "log only".
+if [[ ! -f /etc/vanguard/alert-webhook ]]; then
+  install -d -m 0750 /etc/vanguard
+  install -m 0640 -o root -g 65534 /dev/null /etc/vanguard/alert-webhook
+fi
 install -d -m 0750 /etc/vanguard
 install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
 install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
 systemctl daemon-reload
-systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service
+systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer
 
 echo "== sysctl"
 cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'