From 6f77764fec8b5a542f09b3bd8c4d0b9e52f31bb0 Mon Sep 17 00:00:00 2001 From: Brandyn / Techy Date: Tue, 3 Nov 2026 16:32:09 -0600 Subject: [PATCH] Propagate this build's lessons into the scripts and every runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild sets it up like every other host instead of leaving it to be discovered missing. HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply, and the plan's risks section carries the same warning plus the path-prefix cascade. LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than half-configured; the old Afterwards section is relettered N. Co-Authored-By: Claude Opus 5 (1M context) --- CommunityDiscordMedia.md | 18 +++++++++++++++--- FRESH-INSTALL.md | 35 +++++++++++++++++++++++++++++------ HOSTNAME-CHANGE.md | 26 ++++++++++++++++++++++++-- LOGGING.md | 14 ++++++++++++++ README.md | 5 +++++ STATUS.md | 2 +- addCommunityMediaServer.sh | 11 +++++++---- 7 files changed, 95 insertions(+), 16 deletions(-) diff --git a/CommunityDiscordMedia.md b/CommunityDiscordMedia.md index 320d44c..8c4b085 100644 --- a/CommunityDiscordMedia.md +++ b/CommunityDiscordMedia.md @@ -344,11 +344,17 @@ Six containers, all loopback, published only through the gated `diag.` host: `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy - [ ] CNAME `diag` → the main host - [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env` -- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run) +- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads + the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will + leave the new name without a site block or a certificate without reporting anything + (HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run. - [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing, `journalctl -u vanguard-deadman` is quiet -- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be - checked without the services running +- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the + ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs + its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file +- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right + `service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors) - [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web @@ -430,6 +436,12 @@ Checklist if adopted: - **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1. - **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only. - **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public. +- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames + back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with + no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added. +- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`) + moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the + deadman's probes all have to follow. LOGGING.md lists the four traps this produced. ## Monthly cost Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free. diff --git a/FRESH-INSTALL.md b/FRESH-INSTALL.md index 9a928c9..8bb74ba 100644 --- a/FRESH-INSTALL.md +++ b/FRESH-INSTALL.md @@ -5,7 +5,7 @@ Everything in order. Scripted steps are commands; manual steps are click lists. README.md (section numbers given). ## 0. Before touching the server -- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `auth`, `watch`, `git`, `bot` → `HOST`. +- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `diag`, `auth`, `watch`, `git`, `bot` → `HOST`. Optional: the bare domain `A` → server IP (it redirects to `https://HOST/`). - The Discord bot lives in its **own** repo (ours `VanguardOrg/VanguardBot`), not in this one. Nothing here needs it to exist; section 7a wires it up afterwards. @@ -41,7 +41,7 @@ build, **firewall** (keep this session open; test a new ssh in a second terminal cd ~/vanguard-stream && ./configure.py --full -Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it +Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`diag`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it redirects; Discord bot token + server ID (optional now, needed for the plugin feed); "is live" webhook (optional). Generates `.env` with every secret, the configs, starts all containers, waits, then runs the post-start fixups (Syncthing GUI auth off, Gitea admin user, Jellyfin plugin). `./configure.py --show` prints every credential @@ -117,6 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro 6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the container. By hand: `./deployBotBundle.sh `; back out with `--rollback`. +## 7b. Diagnostics — logs, metrics, alerts (README §5h) +1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector, + VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE` + behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`). +2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and + show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting. +3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing + that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`. +4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D). + ## 8. Pipeline (README "Deployment") 1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo → `./switchToClone.sh clone` (keeps `.env`, generated configs, plugin zips) → `./configure.py --apply`. @@ -341,8 +351,21 @@ L7. dev$ ssh debian@HOST 'docker logs discord-bot --tail 20' L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. Needs the `bot-users` group (admins need it too, or set BOT_REQUIRE_GROUP= and let the bot's WEB_ALLOWED_GROUPS decide). -## M. Afterwards +## M. Diagnostics (browser at https://diag.BASE/) -M1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"' → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply. -M2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update). -M3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7. +M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"' + → five Up. Vector restarting means a config error: docker logs vector --tail 20. +M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the + stream fields and you should see several distinct ones. +M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat + the deadman watches. +M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?' + → no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it. +M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that + regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty. + +## N. Afterwards + +N1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"' → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply. +N2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update). +N3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7. diff --git a/HOSTNAME-CHANGE.md b/HOSTNAME-CHANGE.md index b3da557..beff11d 100644 --- a/HOSTNAME-CHANGE.md +++ b/HOSTNAME-CHANGE.md @@ -15,7 +15,7 @@ Roles: `main` (gateway page at `/`), `files` (copyparty), `clips`, `sync`, `live | Place | Holds | |---|---| | `Caddyfile` | every site block — the source of truth `configure.py` reads back (`# vanguard:` markers) | -| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` | +| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` | | `mediamtx.yml` | `webrtcAdditionalHosts: [MAIN_HOST]` (needs a MediaMTX restart to take effect) | | `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT). `compose up -d` in the apply recreates the affected containers (seconds each) | | Caddy TLS | certificates for new names are fetched automatically; old ones linger harmlessly | @@ -122,8 +122,30 @@ name under BASE, the outpost host (`auth.`) is unchanged. 5. Discord: re-pin `https://media.BASE/` (members), `https://clips.BASE/` (gamers), `https://files.BASE/` (editors). 6. Old `ovh1` bookmarks stop working (no Caddy block, TLS fails) — point people at the gateway. +## D. Adding a NEW host + +**`./configure.py --apply` cannot do this, and will not complain.** It is stateless by design: it reads the +hostnames back out of the `Caddyfile` (`# vanguard:` markers), so a role that is not in there yet has no +value to write and the site is simply left out. The containers come up, Caddy reloads, and the new name has no +site block and therefore no certificate — which presents as an SSL handshake error, not as an error from the +apply. This cost an hour on 2026-11-03 when `diag.` was added. + +Introduce a new host with **`--full`** (asks for every hostname, defaults from the current config) or with +**`--hosts`**, which is the non-interactive form and takes them positionally: + + ./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG # "" leaves one unset + +After that, `--apply` maintains it like any other host, because the marker is now in the file. + +1. DNS: `CNAME` (or `A`) for the new name → the box; check it resolves before continuing. +2. Add the role to `configure.py`: a `write_caddy` block, a `_REQUIRE_GROUP` default in `write_env`, + the prompt in `full()`, the positional in `--hosts`, and the line in `show()`. +3. `./configure.py --hosts …` with every hostname including the new one. +4. Verify the marker landed: `grep -c "vanguard:" Caddyfile` → 1. Then `curl -sI https://NEW/` → 302. +5. Add it to `addCommunityMediaServer.sh` so a rebuild asks for it too. + ## Checklist after any change - dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done + dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE diag.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done https://HOST/whoami → your identity (gate works on the new name) https://watch.BASE/sso/OID/start/authentik → Jellyfin SSO round trip https://git.BASE/user/login → "Sign in with authentik" round trip diff --git a/LOGGING.md b/LOGGING.md index 2be1831..c4a3d92 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -108,6 +108,20 @@ a signal. Nothing inside the stack can report its own death; this can. a working configuration, not an error: `configure.py` generates `observability/alertmanager.yml` with the Discord notifier omitted, so nothing is half-configured and nothing fails to load. +## Gotchas this build produced (all fixed; here so they are not rediscovered) + +- **`-http.pathPrefix` applies to EVERY endpoint, not just the UI.** With VictoriaLogs behind `/logs`, the + ingestion path is `/logs/insert/elasticsearch/`, the health probe is `/logs/health` and the scrape is + `/logs/metrics`. It rejects a write to the unprefixed path with a message naming the missing flag. +- **Every proxied UI needs its own prefix set**, or it loads at the sub-path and then fetches its assets from + the domain root and renders broken. VictoriaMetrics and vmalert use `-http.pathPrefix`; Alertmanager uses + `--web.route-prefix` plus `--web.external-url`. Setting those also moves their metrics and health paths. +- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped; + earlier entries are still on the host for `journalctl --boot=-1`. +- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a + file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it. +- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D. + ## Wiring sketches (the options that were weighed) - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container diff --git a/README.md b/README.md index d3defdf..5dab4a2 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,10 @@ **Hostnames** (all CNAMEs to the main host; one Discord login covers all of them): +> Adding a host later needs `./configure.py --full` or `--hosts`, **not** `--apply`: apply reads the hostnames +> back out of the Caddyfile, so a role that is not in there yet is silently left out and the new name ends up +> with no certificate. HOSTNAME-CHANGE.md §D has the procedure. + | Host | What | Who | |---|---|---| | `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone | @@ -14,6 +18,7 @@ | `clips.BASE` | self-service clip intake (pair your Syncthing) | any member | | `sync.BASE` | Syncthing admin UI | media-admins | | `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) | +| `diag.BASE` | logs, metrics, alerts (§5h) | media-admins | | `watch.BASE` | Jellyfin | Discord via OIDC | | `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — | diff --git a/STATUS.md b/STATUS.md index 6565465..c090491 100644 --- a/STATUS.md +++ b/STATUS.md @@ -59,7 +59,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | normalize sweep | editing standard, one copy per clip | live | | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live | | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) | -| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **committed 2026-11-03, deploy pending**: needs the `diag` CNAME and `ALERT_WEBHOOK_URL`; expect one round of corrections to Vector's journald fields and the LogsQL rules | +| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **deploying 2026-11-03**: containers up; four first-run corrections applied (systemd 257 rejects `current_boot_only: false`; `-http.pathPrefix` moves the ingestion path too; each proxied UI needs its own prefix; a Caddy `respond` body eats `{...}`). `diag.` needed `--hosts`, not `--apply` — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No `ALERT_WEBHOOK_URL` yet, which is a supported state: alerts evaluate and show at `/alerts` but are not delivered | | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** | | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) | | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live | diff --git a/addCommunityMediaServer.sh b/addCommunityMediaServer.sh index 438416a..d854998 100755 --- a/addCommunityMediaServer.sh +++ b/addCommunityMediaServer.sh @@ -39,11 +39,13 @@ SYNC_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("sync","" SYNC=$(ask "Syncthing UI hostname" "${SYNC_CUR:-sync.$BASE}") CLIPS_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("clips",""))') CLIPS=$(ask "Clips self-service hostname" "${CLIPS_CUR:-clips.$BASE}") +DIAG_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("diag",""))') +DIAG=$(ask "Diagnostics hostname: logs, metrics, alerts" "${DIAG_CUR:-diag.$BASE}") # ---- DNS sanity ---------------------------------------------------------------------------------- say "DNS check" MAIN_IP=$(ip_of "$MAIN"); ok=1 -for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS"; do +for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG"; do ip=$(ip_of "$h") if [[ -z $ip ]]; then echo " $h: does not resolve yet"; ok=0 elif [[ $ip != "$MAIN_IP" ]]; then echo " $h -> $ip (main is $MAIN_IP) — points somewhere else"; ok=0 @@ -56,7 +58,7 @@ fi # ---- write configs, .env, start containers ----------------------------------------------------- say "Applying (configure.py --hosts)" -VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" | sed -n '/== writing/,$p' | grep -vE '^ (techy|diceofire|[a-z0-9_-]+) password' || true +VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG" | sed -n '/== writing/,$p' | grep -vE '^ (techy|diceofire|[a-z0-9_-]+) password' || true [[ $DRY == 1 ]] && { echo "(dry run: skipping container wait)"; exit 0; } # ---- wait --------------------------------------------------------------------------------------- @@ -156,14 +158,15 @@ PHASE 3 — Jellyfin (direct play only on this CPU) Custom CSS: paste jellyfin/branding.css -> Save; reload https://$WATCH/ -AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $BOT) — once: +AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $DIAG, $BOT) — once: 1. Applications -> Outposts -> edit "authentik Embedded Outpost" -> config: authentik_host: https://$AUTH 2. Applications -> Create with provider -> Application "Vanguard sites"; Proxy Provider, authorization flow implicit-consent, mode "Forward auth (domain level)", authentication URL https://$AUTH, cookie domain ${BASE}, Invalidation flow default-provider-invalidation-flow (logout = full re-login) -> then edit the Embedded Outpost -> Applications: add it -> Update 3. Per-host access is enforced by Caddy from X-Authentik-Groups: .env BOT_REQUIRE_GROUP=bot-users - (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member). ./configure.py apply regenerates. + (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member), DIAG_REQUIRE_GROUP=media-admins. + ./configure.py apply regenerates. 4. Discord roles -> Authentik groups: Directory -> Groups: create bot-users (keep streamers/editors/admins); Discord source -> Scopes: "guilds guilds.members.read"; Customization -> Policies -> Expression Policy "discord-roles-to-groups" from authentik/discord-roles-to-groups.py (server id + role-id->group table);