techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs follow the Elastic swap, and apply removes orphaned containers

Browse Source 
README 5h, Phase 13, FRESH-INSTALL 7b and appendix M, and STATUS all describe four Elastic containers with
Kibana as the single pane instead of the Victoria shape. Phase 13 keeps the wrong turn on the record: the
first build had no dashboards, which was the point of the exercise, and Grafana was rejected for this purpose
because it is a UI only and its stack is the same piece count with more parts.

configure.py apply now passes --remove-orphans, which is how the eight containers the compose file no longer
lists actually go away rather than lingering.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 16:58:08 -06:00
parent 6f7fb29ae0
commit ef357c6d82
5 changed files with 95 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@@ -2,7 +2,7 @@
> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
> **LOGGING.md** = log framework: journald + rotated Docker logs, container labels, logfmt events, categories; store (Loki/VictoriaLogs/Netdata) chosen later.
> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
@@ -429,40 +429,60 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
Jellyfin and Gitea use OIDC providers instead, not this.
## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`)
## 5h. Diagnostics: Kibana at `diag.HOST`
Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`,
`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands.
Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
dashboards, and alerting, with no sub-paths and no index page of links.
https://diag.HOST/logs logs (VictoriaLogs; 30 days)
https://diag.HOST/metrics-db metrics (VictoriaMetrics; 90 days)
https://diag.HOST/alerts what is firing (Alertmanager)
https://diag.HOST/rules-metrics metric rules · /rules-logs log rules
| container | job |
|---|---|
| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
| `filebeat` | every container's logs (autodiscover) plus our journal units |
| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not
alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart
loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures,
publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook,
`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file.
**Running without a webhook is a supported state**, not a half-configured one: `configure.py` generates
`observability/alertmanager.yml` with the Discord notifier left out entirely, so alerts still evaluate, group
and appear at `/alerts`, they are simply not delivered. The deadman behaves the same way, logging to the
journal and exiting non-zero when it finds something it could not report. Set the variable, re-apply, done.
Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
Four things are pinned on purpose, each because the default bites:
| setting | why |
|---|---|
| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
`http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
correct rather than a bug.
3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
evaluate and show in Kibana, they are simply not delivered.
**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole
vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot
report its own death, and silence looks exactly like health. It reports on change only, announces recovery
once, and without a webhook still logs and exits non-zero so the unit's state is the signal.
nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
| it checks | because |
|---|---|
| Elasticsearch answers, cluster not red | no store, no anything |
| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
| Kibana is available | it is the only UI |
| each Beat wrote a document recently | "running" is not "shipping" |
| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
systemctl list-timers vanguard-deadman.timer
journalctl -u vanguard-deadman -n 20
sudo /usr/local/bin/vanguard-deadman.sh # run it now
sudo /usr/local/bin/vanguard-deadman.sh # run it now; exit 1 means it found something undeliverable
Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker
socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen
deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the
journald log driver on every container or a read-only socket proxy in front of vector.
Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
correlation above work.
## 6. Editors