Commit Graph

8 Commits

Author SHA1 Message Date
d3ff452546 Generate alertmanager.yml so no-webhook is a valid state, not a broken one
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty
webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord
notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still
evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in
/etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the
container's user while the repo copy cannot.

alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 16:01:49 -06:00
cea02d2ee8 Phase 13 integration: install.sh, configure.py, and a deadman that degrades cleanly
install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.

configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.

The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.

check.sh validates all five observability configs; confirmed it catches a deliberate break.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:56:49 -06:00
09882833a0 Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack
Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions
only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four
containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:35:39 -06:00
3662eb3465 Phase 14: diag. host with VictoriaLogs at /logs, ops controls later
Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:33:47 -06:00
1e4c0e88dc The bot's secrets live in Gitea: signed settings travel with the release bundle
POST /deploy-bot may now carry X-Vanguard-Env, base64 JSON of the bot's own settings, and the HMAC covers
that header together with the body so the pair cannot be separated or swapped. dumpd merges only the keys
BOT_ENV_KEYS allows (DISCORD_BOT_TOKEN, DISCORD_GUILD_ID by default) into the stack .env, atomically and
preserving mode and owner, and reports the rest to the CI job as settings_refused. The server keeps what it
generates: a CI job cannot touch the database password or the Authentik keys.

Verified: allowed keys applied, server-owned keys refused while the rest still applied, a settings swap
after signing rejected with 401, and an empty settings header leaving .env untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:37:35 -06:00
b05f129bed Bot releases install themselves: signed bundle upload to dumpd, no Docker socket in CI
dumpd gains POST /deploy-bot on the bridge-only auth port: HMAC-SHA256 over the body with BOT_DEPLOY_SECRET,
a 300 MB cap, one install at a time, and the installer's output returned in the response so the CI job shows
it. BOT_DEPLOY_AUTO=0 makes the server refuse. configure.py generates the secret, writes both settings into
/etc/vanguard/deploy.env, installs deployBotBundle.sh as /usr/local/bin/vanguard-deploy-bot and creates
/srv/bot/app; --show prints what to paste into Gitea.

Chose this over mounting the Docker socket into job containers, which would give every CI job root on the
host. Also moved threading/subprocess to dumpd's module imports, which the new module-level lock needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-02 21:32:34 -06:00
855651c5a4 Metrics taps: node_exporter + cAdvisor containers, MediaMTX and Caddy metrics on loopback (Prometheus format, store-agnostic)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:55:42 -06:00
704bc8c4d3 Logging framework: logfmt events in our scripts, container labels + rotated json-file logs, Caddy JSON access logs; LOGGING.md
Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:53:05 -06:00