Docs follow the Elastic swap, and apply removes orphaned containers
README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose because it is a UI only and its stack is the same piece count with more parts. configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer lists actually go away rather than lingering. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -117,15 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro
|
||||
6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
|
||||
container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
|
||||
|
||||
## 7b. Diagnostics — logs, metrics, alerts (README §5h)
|
||||
1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector,
|
||||
VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE`
|
||||
behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
|
||||
2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and
|
||||
show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting.
|
||||
3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing
|
||||
that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`.
|
||||
4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D).
|
||||
## 7b. Diagnostics — Kibana (README §5h)
|
||||
1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND
|
||||
metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published
|
||||
except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
|
||||
2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it.
|
||||
3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still
|
||||
logs; only delivery is missing.
|
||||
4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the
|
||||
always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them.
|
||||
5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D).
|
||||
|
||||
## 8. Pipeline (README "Deployment")
|
||||
1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
|
||||
@@ -353,16 +354,17 @@ L8. Private window → https://bot.BASE/ → Discord login → the bot's pages.
|
||||
|
||||
## M. Diagnostics (browser at https://diag.BASE/)
|
||||
|
||||
M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"'
|
||||
→ five Up. Vector restarting means a config error: docker logs vector --tail 20.
|
||||
M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the
|
||||
stream fields and you should see several distinct ones.
|
||||
M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat
|
||||
the deadman watches.
|
||||
M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
|
||||
→ no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it.
|
||||
M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that
|
||||
regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty.
|
||||
M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"'
|
||||
→ four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap.
|
||||
M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting.
|
||||
M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs
|
||||
and metrics. Metricbeat also installed its own system and Docker dashboards: start from those.
|
||||
M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible).
|
||||
M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule:
|
||||
always true, every minute, action = webhook POST to http://<docker gateway>:8081/heartbeat
|
||||
(gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}').
|
||||
M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
|
||||
→ silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing.
|
||||
|
||||
## N. Afterwards
|
||||
|
||||
|
||||
Reference in New Issue
Block a user