techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 13 integration: install.sh, configure.py, and a deadman that degrades cleanly

Browse Source 
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.

configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.

The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.

check.sh validates all five observability configs; confirmed it catches a deliberate break.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 15:56:49 -06:00
parent 8ae43905dc
commit cea02d2ee8
8 changed files with 147 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
README.md
View File

@@ -424,6 +424,37 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
Jellyfin and Gitea use OIDC providers instead, not this.
## 5h. Diagnostics: logs, metrics, alerts (`diag.HOST`)
Six containers, all loopback, published only through `diag.HOST` behind the gate (`DIAG_REQUIRE_GROUP`,
`media-admins` by default). `LOGGING.md` is the contract they consume; this is where it lands.
https://diag.HOST/logs logs (VictoriaLogs; 30 days)
https://diag.HOST/metrics-db metrics (VictoriaMetrics; 90 days)
https://diag.HOST/alerts what is firing (Alertmanager)
https://diag.HOST/rules-metrics metric rules · /rules-logs log rules
Alerting is **vmalert** evaluating rules and **Alertmanager** delivering them, because VictoriaLogs does not
alert by itself. Metric rules (`observability/alerts-metrics.yml`) cover disk, load, memory, RAID, restart
loops and a missing core container; log rules (`observability/alerts-logs.yml`, LogsQL) cover sweep failures,
publish-deny bursts, failed bot releases and certificate trouble. Both notify one Discord webhook,
`ALERT_WEBHOOK_URL` in `.env`, written to `/etc/vanguard/alert-webhook` so it never sits in a config file.
**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
nothing but curl: are all six answering, is the always-firing `Watchdog` alert still coming through the whole
vmalert → Alertmanager → dumpd `/heartbeat` path, and are the disks under threshold. An alerting stack cannot
report its own death, and silence looks exactly like health. It reports on change only, announces recovery
once, and without a webhook still logs and exits non-zero so the unit's state is the signal.
systemctl list-timers vanguard-deadman.timer
journalctl -u vanguard-deadman -n 20
sudo /usr/local/bin/vanguard-deadman.sh # run it now
Log collection uses `docker_logs`, which needs the Docker API, so **the vector container holds the Docker
socket and therefore root-equivalent access to this host** — the same exposure act_runner has. Chosen
deliberately (Phase 13 option A). The upgrade paths, neither of which changes anything downstream, are the
journald log driver on every container or a read-only socket proxy in front of vector.
## 6. Editors
Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik