Docs follow the Elastic swap, and apply removes orphaned containers

README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with
Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the
first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose
because it is a UI only and its stack is the same piece count with more parts.

configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer
lists actually go away rather than lingering.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 16:58:08 -06:00
parent 6f7fb29ae0
commit ef357c6d82
5 changed files with 95 additions and 85 deletions

View File

@@ -320,45 +320,31 @@ permanent reference is needed for long sessions, not just a start burst.
export at 2560×1440)
- rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (built; needs DNS + a webhook)
### Phase 13 — `diag.` host: Kibana (logs + metrics + alerting) (built; needs a webhook + two Kibana rules)
**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a
growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two
services and a dashboard tool we would only use as a log viewer). The framework everything feeds is
`LOGGING.md`; as-built detail lives there and in README §5h.
**Decided 2026-11-03, after one wrong turn.** First build was VictoriaLogs + Vector + VictoriaMetrics + two
vmalert instances + Alertmanager, eight containers counting the exporters, and it produced **no dashboards** —
five vendor UIs behind one hostname and an index page of links. The requirement was one tool with views, and
the honest answers to that were Netdata (one container, weakest log search) or Elastic (four containers, one
UI, strongest log search). Elastic was chosen. Grafana was considered and rejected *for this purpose*: it is a
UI only, so its stack is Loki + Prometheus + Alloy + Grafana, which is the same piece count with more parts.
Six containers, all loopback, published only through the gated `diag.` host:
`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`.
Four containers: `elasticsearch` (both logs and metrics) · `kibana` (the whole `diag.` host, at the root) ·
`filebeat` · `metricbeat` (which also retires node_exporter and cAdvisor). Detail in README §5h and LOGGING.md.
- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL`
asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534)
- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose
bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start),
the deadman script and timer
- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so
a critical disk does not also report as filling
- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat
through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery
once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal
- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in
`observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
- [ ] CNAME `diag` → the main host
- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads
the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will
leave the new name without a site block or a certificate without reporting anything
(HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run.
- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
`journalctl -u vanguard-deadman` is quiet
- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the
ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs
its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file
- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right
`service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors)
- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
page with Docker access, never a generic command runner.
- [x] compose: the four, all loopback, published only through the gated `diag.` host
- [x] pinned: 2 GB heap, `vm.max_map_count`, security off (loopback + Authentik), fixed Kibana encryption key
- [x] both Beats emit the same correlation fields, so one dashboard spans logs and metrics
- [x] `configure.py`: `diag` role → Kibana at the root; `--remove-orphans` on apply so the old eight go away
- [x] `install.sh`: the ES data dir, the sysctl, the deadman
- [x] deadman repointed at Elastic, including the two failures a process check misses: a Beat that runs without
shipping, and indices flipped read-only by the watermark (which never clears itself)
- [ ] a Discord webhook → `ALERT_WEBHOOK_URL`
- [ ] in Kibana: the retention delete phase; the always-firing heartbeat rule → dumpd `/heartbeat`; a webhook
connector for alerts; then rules for the conditions the old vmalert files listed (disk, load, restart
loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble)
- [ ] later: ops controls on this host (restart buttons) — allowlisted dumpd endpoints, authorised from the
Authentik headers, every invocation logged as a logfmt event. Never a page with Docker access.
### Phase 14 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
- server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved

View File

@@ -117,15 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro
6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
## 7b. Diagnostics — logs, metrics, alerts (README §5h)
1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector,
VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE`
behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and
show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting.
3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing
that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`.
4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D).
## 7b. Diagnostics — Kibana (README §5h)
1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND
metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published
except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it.
3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still
logs; only delivery is missing.
4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the
always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them.
5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D).
## 8. Pipeline (README "Deployment")
1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
@@ -353,16 +354,17 @@ L8. Private window → https://bot.BASE/ → Discord login → the bot's pages.
## M. Diagnostics (browser at https://diag.BASE/)
M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"'
→ five Up. Vector restarting means a config error: docker logs vector --tail 20.
M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the
stream fields and you should see several distinct ones.
M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat
the deadman watches.
M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
→ no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it.
M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that
regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty.
M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"'
→ four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap.
M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting.
M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs
and metrics. Metricbeat also installed its own system and Docker dashboards: start from those.
M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible).
M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule:
always true, every minute, action = webhook POST to http://<docker gateway>:8081/heartbeat
(gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}').
M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
→ silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing.
## N. Afterwards

View File

@@ -2,7 +2,7 @@
> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
> **LOGGING.md** = log framework: journald + rotated Docker logs, container labels, logfmt events, categories; store (Loki/VictoriaLogs/Netdata) chosen later.
> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
@@ -429,40 +429,60 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
Jellyfin and Gitea use OIDC providers instead, not this.
## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`)
## 5h. Diagnostics: Kibana at `diag.HOST`
Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`,
`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands.
Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
dashboards, and alerting, with no sub-paths and no index page of links.
https://diag.HOST/logs logs (VictoriaLogs; 30 days)
https://diag.HOST/metrics-db metrics (VictoriaMetrics; 90 days)
https://diag.HOST/alerts what is firing (Alertmanager)
https://diag.HOST/rules-metrics metric rules · /rules-logs log rules
| container | job |
|---|---|
| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
| `filebeat` | every container's logs (autodiscover) plus our journal units |
| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not
alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart
loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures,
publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook,
`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file.
**Running without a webhook is a supported state**, not a half-configured one: `configure.py` generates
`observability/alertmanager.yml` with the Discord notifier left out entirely, so alerts still evaluate, group
and appear at `/alerts`, they are simply not delivered. The deadman behaves the same way, logging to the
journal and exiting non-zero when it finds something it could not report. Set the variable, re-apply, done.
Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
Four things are pinned on purpose, each because the default bites:
| setting | why |
|---|---|
| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
`http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
correct rather than a bug.
3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
evaluate and show in Kibana, they are simply not delivered.
**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole
vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot
report its own death, and silence looks exactly like health. It reports on change only, announces recovery
once, and without a webhook still logs and exits non-zero so the unit's state is the signal.
nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
| it checks | because |
|---|---|
| Elasticsearch answers, cluster not red | no store, no anything |
| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
| Kibana is available | it is the only UI |
| each Beat wrote a document recently | "running" is not "shipping" |
| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
systemctl list-timers vanguard-deadman.timer
journalctl -u vanguard-deadman -n 20
sudo /usr/local/bin/vanguard-deadman.sh # run it now
sudo /usr/local/bin/vanguard-deadman.sh # run it now; exit 1 means it found something undeliverable
Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker
socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen
deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the
journald log driver on every container or a read-only socket proxy in front of vector.
Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
correlation above work.
## 6. Editors

View File

@@ -59,9 +59,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **deploying 2026-11-03**: containers up; four first-run corrections applied (systemd 257 rejects `current_boot_only: false`; `-http.pathPrefix` moves the ingestion path too; each proxied UI needs its own prefix; a Caddy `respond` body eats `{...}`). `diag.` needed `--hosts`, not `--apply` — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No `ALERT_WEBHOOK_URL` yet, which is a supported state: alerts evaluate and show at `/alerts` but are not delivered |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) |
| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
| Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |

View File

@@ -584,7 +584,9 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
run(["sudo", "systemctl", "restart", "dumpd"])
print("== containers")
dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"]
run(dc + ["up", "-d", "--build"]) # recreates only services whose compose definition changed
# --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how
# the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them.
run(dc + ["up", "-d", "--build", "--remove-orphans"]) # recreates only services whose definition changed
run(dc + ["restart", "copyparty"]) # copyparty re-reads its config on start
run(dc + ["exec", "caddy", "caddy", "reload", "--config", "/etc/caddy/Caddyfile"]) # zero-downtime
post_start_fixups(dc)