Phase 13 build: logs, metrics, alerting and a deadman on a diag. host

Six containers, all loopback-published, reachable only through the gated diag. host:
  victorialogs      30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
  vector            docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
  victoriametrics   90d, scrapes the four taps that were already live plus the new services
  vmalert-metrics   11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
  vmalert-logs      5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
  alertmanager      groups and inhibits, posts to Discord, webhook read from a 0600 file not the config

The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.

Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 15:45:58 -06:00
parent 09882833a0
commit 8ae43905dc
13 changed files with 695 additions and 63 deletions

View File

@@ -153,6 +153,67 @@ clips.example.com {
}
}
# vanguard:diag
diag.example.com {
encode zstd gzip
log {
output stderr
format filter {
wrap json
fields {
request>headers>Cookie delete
request>headers>Authorization delete
request>headers>X-Stream-Auth delete
}
}
}
# /logout on every gated host = sign out of the Discord login (domain-wide session)
redir /logout /outpost.goauthentik.io/sign_out 302
# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
forward_auth 127.0.0.1:9000 {
uri /outpost.goauthentik.io/auth/caddy
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
trusted_proxies private_ranges
}
@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
handle @lacks_group {
respond "This page is for members of the 'media-admins' group." 403
}
# VictoriaLogs runs with -http.pathPrefix=/logs and emits /logs/... URLs itself, so the prefix is
# passed through unchanged. Stripping it would break the UI's own asset paths.
redir /logs /logs/select/vmui/ 302
redir /logs/ /logs/select/vmui/ 302
handle /logs* {
reverse_proxy 127.0.0.1:9428
}
handle /metrics-db* {
reverse_proxy 127.0.0.1:8428 # VictoriaMetrics UI + query API
}
handle /alerts* {
reverse_proxy 127.0.0.1:9093 # Alertmanager: what is firing, silences
}
handle /rules-metrics* {
reverse_proxy 127.0.0.1:8880 # vmalert over VictoriaMetrics
}
handle /rules-logs* {
reverse_proxy 127.0.0.1:8881 # vmalert over VictoriaLogs
}
handle {
respond "Vanguard diagnostics
/logs logs (VictoriaLogs)
/metrics-db metrics (VictoriaMetrics)
/alerts what is firing (Alertmanager)
/rules-metrics metric rules (vmalert)
/rules-logs log rules (vmalert)
Alerts go to Discord. The deadman runs on the host and watches this stack." 200
}
}
# vanguard:live
live.example.com {
encode zstd gzip