Docs follow the Elastic swap, and apply removes orphaned containers

README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with
Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the
first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose
because it is a UI only and its stack is the same piece count with more parts.

configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer
lists actually go away rather than lingering.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -320,45 +320,31 @@ permanent reference is needed for long sessions, not just a start burst.
       export at 2560×1440)
 - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
 
-### Phase 13 — `diag.` host: logs, metrics, alerting, deadman  (built; needs DNS + a webhook)
+### Phase 13 — `diag.` host: Kibana (logs + metrics + alerting)  (built; needs a webhook + two Kibana rules)
 
-**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a
+**Decided 2026-11-03, after one wrong turn.** First build was VictoriaLogs + Vector + VictoriaMetrics + two
-growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two
+vmalert instances + Alertmanager, eight containers counting the exporters, and it produced **no dashboards** —
-services and a dashboard tool we would only use as a log viewer). The framework everything feeds is
+five vendor UIs behind one hostname and an index page of links. The requirement was one tool with views, and
-`LOGGING.md`; as-built detail lives there and in README §5h.
+the honest answers to that were Netdata (one container, weakest log search) or Elastic (four containers, one
+UI, strongest log search). Elastic was chosen. Grafana was considered and rejected *for this purpose*: it is a
+UI only, so its stack is Loki + Prometheus + Alloy + Grafana, which is the same piece count with more parts.
 
-Six containers, all loopback, published only through the gated `diag.` host:
+Four containers: `elasticsearch` (both logs and metrics) · `kibana` (the whole `diag.` host, at the root) ·
-`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`.
+`filebeat` · `metricbeat` (which also retires node_exporter and cAdvisor). Detail in README §5h and LOGGING.md.
 
-- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL`
+- [x] compose: the four, all loopback, published only through the gated `diag.` host
-      asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534)
+- [x] pinned: 2 GB heap, `vm.max_map_count`, security off (loopback + Authentik), fixed Kibana encryption key
-- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose
+- [x] both Beats emit the same correlation fields, so one dashboard spans logs and metrics
-      bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start),
+- [x] `configure.py`: `diag` role → Kibana at the root; `--remove-orphans` on apply so the old eight go away
-      the deadman script and timer
+- [x] `install.sh`: the ES data dir, the sysctl, the deadman
-- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so
+- [x] deadman repointed at Elastic, including the two failures a process check misses: a Beat that runs without
-      a critical disk does not also report as filling
+      shipping, and indices flipped read-only by the watermark (which never clears itself)
-- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat
+- [ ] a Discord webhook → `ALERT_WEBHOOK_URL`
-      through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery
+- [ ] in Kibana: the retention delete phase; the always-firing heartbeat rule → dumpd `/heartbeat`; a webhook
-      once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal
+      connector for alerts; then rules for the conditions the old vmalert files listed (disk, load, restart
-- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in
+      loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble)
-      `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
+- [ ] later: ops controls on this host (restart buttons) — allowlisted dumpd endpoints, authorised from the
-- [ ] CNAME `diag` → the main host
+      Authentik headers, every invocation logged as a logfmt event. Never a page with Docker access.
-- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
-- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads
-      the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will
-      leave the new name without a site block or a certificate without reporting anything
-      (HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run.
-- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
-      `journalctl -u vanguard-deadman` is quiet
-- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the
-      ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs
-      its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file
-- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right
-      `service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors)
-- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
-      way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
-      invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
-      page with Docker access, never a generic command runner.
 
 ### Phase 14 — push-to-deploy  (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
 - server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved

FRESH-INSTALL.md

@@ -117,15 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro
 6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
    container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
 
-## 7b. Diagnostics — logs, metrics, alerts (README §5h)
+## 7b. Diagnostics — Kibana (README §5h)
-1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector,
+1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND
-   VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE`
+   metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published
-   behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
+   except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
-2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and
+2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it.
-   show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting.
+3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still
-3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing
+   logs; only delivery is missing.
-   that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`.
+4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the
-4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D).
+   always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them.
+5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D).
 
 ## 8. Pipeline (README "Deployment")
 1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
@@ -353,16 +354,17 @@ L8. Private window → https://bot.BASE/ → Discord login → the bot's pages.
 
 ## M. Diagnostics (browser at https://diag.BASE/)
 
-M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"'
+M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"'
-    → five Up. Vector restarting means a config error: docker logs vector --tail 20.
+    → four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap.
-M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the
+M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting.
-    stream fields and you should see several distinct ones.
+M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs
-M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat
+    and metrics. Metricbeat also installed its own system and Docker dashboards: start from those.
-    the deadman watches.
+M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible).
-M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule:
-    → no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it.
+    always true, every minute, action = webhook POST to http://<docker gateway>:8081/heartbeat
-M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that
+    (gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}').
-    regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty.
+M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+    → silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing.
 
 ## N. Afterwards
 

README.md

@@ -2,7 +2,7 @@
 
 > **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
 > **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
-> **LOGGING.md** = log framework: journald + rotated Docker logs, container labels, logfmt events, categories; store (Loki/VictoriaLogs/Netdata) chosen later.
+> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
 > **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
 
 **Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
@@ -429,40 +429,60 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
 
 Jellyfin and Gitea use OIDC providers instead, not this.
 
-## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`)
+## 5h. Diagnostics: Kibana at `diag.HOST`
 
-Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`,
+Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
-`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands.
+dashboards, and alerting, with no sub-paths and no index page of links.
 
-    https://diag.HOST/logs            logs (VictoriaLogs; 30 days)
+| container | job |
-    https://diag.HOST/metrics-db      metrics (VictoriaMetrics; 90 days)
+|---|---|
-    https://diag.HOST/alerts          what is firing (Alertmanager)
+| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
-    https://diag.HOST/rules-metrics   metric rules · /rules-logs  log rules
+| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
+| `filebeat` | every container's logs (autodiscover) plus our journal units |
+| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
 
-Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not
+Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
-alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart
+`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
-loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures,
+narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
-publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook,
+
-`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file.
+Four things are pinned on purpose, each because the default bites:
-**Running without a webhook is a supported state**, not a half-configured one: `configure.py` generates
+
-`observability/alertmanager.yml` with the Discord notifier left out entirely, so alerts still evaluate, group
+| setting | why |
-and appear at `/alerts`, they are simply not delivered. The deadman behaves the same way, logging to the
+|---|---|
-journal and exiting non-zero when it finds something it could not report. Set the variable, re-apply, done.
+| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
+| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
+| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
+| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
+
+**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
+
+1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
+2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
+   `http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
+   deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
+   correct rather than a bug.
+3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
+   evaluate and show in Kibana, they are simply not delivered.
 
 **The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
-nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole
+nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
-vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot
+
-report its own death, and silence looks exactly like health. It reports on change only, announces recovery
+| it checks | because |
-once, and without a webhook still logs and exits non-zero so the unit's state is the signal.
+|---|---|
+| Elasticsearch answers, cluster not red | no store, no anything |
+| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
+| Kibana is available | it is the only UI |
+| each Beat wrote a document recently | "running" is not "shipping" |
+| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
+| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
 
     systemctl list-timers vanguard-deadman.timer
     journalctl -u vanguard-deadman -n 20
-    sudo /usr/local/bin/vanguard-deadman.sh          # run it now
+    sudo /usr/local/bin/vanguard-deadman.sh          # run it now; exit 1 means it found something undeliverable
 
-Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker
+Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
-socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen
+act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
-deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the
+correlation above work.
-journald log driver on every container or a read-only socket proxy in front of vector.
 
 ## 6. Editors
 

STATUS.md

@@ -59,9 +59,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
-| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **deploying 2026-11-03**: containers up; four first-run corrections applied (systemd 257 rejects `current_boot_only: false`; `-http.pathPrefix` moves the ingestion path too; each proxied UI needs its own prefix; a Caddy `respond` body eats `{...}`). `diag.` needed `--hosts`, not `--apply` — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No `ALERT_WEBHOOK_URL` yet, which is a supported state: alerts evaluate and show at `/alerts` but are not delivered |
+| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
-| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
+| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
-| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) |
+| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
 | Jellyfin | watch page, SSO via Authentik, direct play only | live |
 | Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |

configure.py

@@ -584,7 +584,9 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     run(["sudo", "systemctl", "restart", "dumpd"])
     print("== containers")
     dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"]
-    run(dc + ["up", "-d", "--build"])                      # recreates only services whose compose definition changed
+    # --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how
+    # the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them.
+    run(dc + ["up", "-d", "--build", "--remove-orphans"])   # recreates only services whose definition changed
     run(dc + ["restart", "copyparty"])                      # copyparty re-reads its config on start
     run(dc + ["exec", "caddy", "caddy", "reload", "--config", "/etc/caddy/Caddyfile"])   # zero-downtime
     post_start_fixups(dc)