diff --git a/CommunityDiscordMedia.md b/CommunityDiscordMedia.md index 8c4b085..af2bb2d 100644 --- a/CommunityDiscordMedia.md +++ b/CommunityDiscordMedia.md @@ -320,45 +320,31 @@ permanent reference is needed for long sessions, not just a start burst. export at 2560×1440) - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode) -### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (built; needs DNS + a webhook) +### Phase 13 — `diag.` host: Kibana (logs + metrics + alerting) (built; needs a webhook + two Kibana rules) -**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a -growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two -services and a dashboard tool we would only use as a log viewer). The framework everything feeds is -`LOGGING.md`; as-built detail lives there and in README §5h. +**Decided 2026-11-03, after one wrong turn.** First build was VictoriaLogs + Vector + VictoriaMetrics + two +vmalert instances + Alertmanager, eight containers counting the exporters, and it produced **no dashboards** — +five vendor UIs behind one hostname and an index page of links. The requirement was one tool with views, and +the honest answers to that were Netdata (one container, weakest log search) or Elastic (four containers, one +UI, strongest log search). Elastic was chosen. Grafana was considered and rejected *for this purpose*: it is a +UI only, so its stack is Loki + Prometheus + Alloy + Grafana, which is the same piece count with more parts. -Six containers, all loopback, published only through the gated `diag.` host: -`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`. +Four containers: `elasticsearch` (both logs and metrics) · `kibana` (the whole `diag.` host, at the root) · +`filebeat` · `metricbeat` (which also retires node_exporter and cAdvisor). Detail in README §5h and LOGGING.md. -- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL` - asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534) -- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose - bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start), - the deadman script and timer -- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so - a critical disk does not also report as filling -- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat - through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery - once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal -- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in - `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy -- [ ] CNAME `diag` → the main host -- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env` -- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads - the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will - leave the new name without a site block or a certificate without reporting anything - (HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run. -- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing, - `journalctl -u vanguard-deadman` is quiet -- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the - ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs - its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file -- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right - `service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors) -- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong - way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every - invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web - page with Docker access, never a generic command runner. +- [x] compose: the four, all loopback, published only through the gated `diag.` host +- [x] pinned: 2 GB heap, `vm.max_map_count`, security off (loopback + Authentik), fixed Kibana encryption key +- [x] both Beats emit the same correlation fields, so one dashboard spans logs and metrics +- [x] `configure.py`: `diag` role → Kibana at the root; `--remove-orphans` on apply so the old eight go away +- [x] `install.sh`: the ES data dir, the sysctl, the deadman +- [x] deadman repointed at Elastic, including the two failures a process check misses: a Beat that runs without + shipping, and indices flipped read-only by the watermark (which never clears itself) +- [ ] a Discord webhook → `ALERT_WEBHOOK_URL` +- [ ] in Kibana: the retention delete phase; the always-firing heartbeat rule → dumpd `/heartbeat`; a webhook + connector for alerts; then rules for the conditions the old vmalert files listed (disk, load, restart + loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble) +- [ ] later: ops controls on this host (restart buttons) — allowlisted dumpd endpoints, authorised from the + Authentik headers, every invocation logged as a logfmt event. Never a page with Docker access. ### Phase 14 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/) - server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved diff --git a/FRESH-INSTALL.md b/FRESH-INSTALL.md index 8bb74ba..dae9d66 100644 --- a/FRESH-INSTALL.md +++ b/FRESH-INSTALL.md @@ -117,15 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro 6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the container. By hand: `./deployBotBundle.sh `; back out with `--rollback`. -## 7b. Diagnostics — logs, metrics, alerts (README §5h) -1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector, - VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE` - behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`). -2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and - show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting. -3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing - that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`. -4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D). +## 7b. Diagnostics — Kibana (README §5h) +1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND + metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published + except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`). +2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it. +3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still + logs; only delivery is missing. +4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the + always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them. +5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D). ## 8. Pipeline (README "Deployment") 1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo @@ -353,16 +354,17 @@ L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. ## M. Diagnostics (browser at https://diag.BASE/) -M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"' - → five Up. Vector restarting means a config error: docker logs vector --tail 20. -M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the - stream fields and you should see several distinct ones. -M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat - the deadman watches. -M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?' - → no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it. -M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that - regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty. +M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"' + → four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap. +M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting. +M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs + and metrics. Metricbeat also installed its own system and Docker dashboards: start from those. +M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible). +M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule: + always true, every minute, action = webhook POST to http://:8081/heartbeat + (gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'). +M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?' + → silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing. ## N. Afterwards diff --git a/README.md b/README.md index 5dab4a2..600c23e 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included. > **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely. -> **LOGGING.md** = log framework: journald + rotated Docker logs, container labels, logfmt events, categories; store (Loki/VictoriaLogs/Netdata) chosen later. +> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h). > **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app). **Hostnames** (all CNAMEs to the main host; one Discord login covers all of them): @@ -429,40 +429,60 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe Jellyfin and Gitea use OIDC providers instead, not this. -## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`) +## 5h. Diagnostics: Kibana at `diag.HOST` -Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`, -`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands. +Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover, +dashboards, and alerting, with no sub-paths and no index page of links. - https://diag.HOST/logs logs (VictoriaLogs; 30 days) - https://diag.HOST/metrics-db metrics (VictoriaMetrics; 90 days) - https://diag.HOST/alerts what is firing (Alertmanager) - https://diag.HOST/rules-metrics metric rules · /rules-logs log rules +| container | job | +|---|---| +| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only | +| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate | +| `filebeat` | every container's logs (autodiscover) plus our journal units | +| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints | -Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not -alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart -loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures, -publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook, -`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file. -**Running without a webhook is a supported state**, not a half-configured one: `configure.py` generates -`observability/alertmanager.yml` with the Discord notifier left out entirely, so alerts still evaluate, group -and appear at `/alerts`, they are simply not delivered. The deadman behaves the same way, logging to the -journal and exiting non-zero when it finds something it could not report. Set the variable, re-apply, done. +Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`, +`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter +narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work. + +Four things are pinned on purpose, each because the default bites: + +| setting | why | +|---|---| +| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 | +| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it | +| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login | +| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart | + +**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy): + +1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase. +2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to + `http://:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the + deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is + correct rather than a bug. +3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still + evaluate and show in Kibana, they are simply not delivered. **The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with -nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole -vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot -report its own death, and silence looks exactly like health. It reports on change only, announces recovery -once, and without a webhook still logs and exits non-zero so the unit's state is the signal. +nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke: + +| it checks | because | +|---|---| +| Elasticsearch answers, cluster not red | no store, no anything | +| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead | +| Kibana is available | it is the only UI | +| each Beat wrote a document recently | "running" is not "shipping" | +| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path | +| the disks | the likeliest death, and it must be reported even when the reporter is what broke | systemctl list-timers vanguard-deadman.timer journalctl -u vanguard-deadman -n 20 - sudo /usr/local/bin/vanguard-deadman.sh # run it now + sudo /usr/local/bin/vanguard-deadman.sh # run it now; exit 1 means it found something undeliverable -Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker -socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen -deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the -journald log driver on every container or a read-only socket proxy in front of vector. +Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure +act_runner has. Accepted deliberately; the alternative is losing the container labels that make the +correlation above work. ## 6. Editors diff --git a/STATUS.md b/STATUS.md index c090491..cfbac58 100644 --- a/STATUS.md +++ b/STATUS.md @@ -59,9 +59,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | normalize sweep | editing standard, one copy per clip | live | | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live | | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) | -| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **deploying 2026-11-03**: containers up; four first-run corrections applied (systemd 257 rejects `current_boot_only: false`; `-http.pathPrefix` moves the ingestion path too; each proxied UI needs its own prefix; a Caddy `respond` body eats `{...}`). `diag.` needed `--hosts`, not `--apply` — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No `ALERT_WEBHOOK_URL` yet, which is a supported state: alerts evaluate and show at `/alerts` but are not delivered | -| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** | -| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) | +| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) | +| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** | +| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending | | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live | | Jellyfin | watch page, SSO via Authentik, direct play only | live | | Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 | diff --git a/configure.py b/configure.py index e03509d..69d377d 100755 --- a/configure.py +++ b/configure.py @@ -584,7 +584,9 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis run(["sudo", "systemctl", "restart", "dumpd"]) print("== containers") dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"] - run(dc + ["up", "-d", "--build"]) # recreates only services whose compose definition changed + # --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how + # the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them. + run(dc + ["up", "-d", "--build", "--remove-orphans"]) # recreates only services whose definition changed run(dc + ["restart", "copyparty"]) # copyparty re-reads its config on start run(dc + ["exec", "caddy", "caddy", "reload", "--config", "/etc/caddy/Caddyfile"]) # zero-downtime post_start_fixups(dc)