techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 13 integration: install.sh, configure.py, and a deadman that degrades cleanly

Browse Source 
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.

configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.

The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.

check.sh validates all five observability configs; confirmed it catches a deliberate break.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 15:56:49 -06:00
parent 8ae43905dc
commit cea02d2ee8
8 changed files with 147 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
LOGGING.md
View File

@@ -84,7 +84,30 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
## Wiring sketches (not deployed; pick one later)
## As built (Phase 13)
| container | port | job |
|---|---|---|
| `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged |
| `vector` | — | `docker_logs` + `journald` → VictoriaLogs (elasticsearch bulk). **Holds the Docker socket** |
| `victoriametrics` | 8428 | 90 d, scrapes the taps above (`observability/scrape.yml`) |
| `vmalert-metrics` | 8880 | `observability/alerts-metrics.yml` against VictoriaMetrics |
| `vmalert-logs` | 8881 | `observability/alerts-logs.yml` (LogsQL) against VictoriaLogs' stats endpoints |
| `alertmanager` | 9093 | groups, inhibits, posts to Discord (`webhook_url_file`, never in the config) |
Published only through `diag.BASE`, gated on `DIAG_REQUIRE_GROUP` (`media-admins`): `/logs`, `/metrics-db`,
`/alerts`, `/rules-metrics`, `/rules-logs`, and an index at `/`.
**The deadman** (`bin/vanguard-deadman.sh`, host timer, 5 min) is deliberately outside all of it: curl only, it
checks each component answers, that the always-firing `Watchdog` alert is still arriving through
vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. It speaks on change, announces recovery once, and
with no webhook configured still logs and exits non-zero so `systemctl is-failed vanguard-deadman.service` is
a signal. Nothing inside the stack can report its own death; this can.
`ALERT_WEBHOOK_URL` in `.env` → `/etc/vanguard/alert-webhook` (0640 root:65534). Empty means log-only, which
is a working configuration, not an error.
## Wiring sketches (the options that were weighed)
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).