Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -153,6 +153,67 @@ clips.example.com {
|
||||
}
|
||||
}
|
||||
|
||||
# vanguard:diag
|
||||
diag.example.com {
|
||||
encode zstd gzip
|
||||
log {
|
||||
output stderr
|
||||
format filter {
|
||||
wrap json
|
||||
fields {
|
||||
request>headers>Cookie delete
|
||||
request>headers>Authorization delete
|
||||
request>headers>X-Stream-Auth delete
|
||||
}
|
||||
}
|
||||
}
|
||||
# /logout on every gated host = sign out of the Discord login (domain-wide session)
|
||||
redir /logout /outpost.goauthentik.io/sign_out 302
|
||||
# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
|
||||
# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
|
||||
# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
|
||||
reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
|
||||
forward_auth 127.0.0.1:9000 {
|
||||
uri /outpost.goauthentik.io/auth/caddy
|
||||
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
|
||||
trusted_proxies private_ranges
|
||||
}
|
||||
@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
|
||||
handle @lacks_group {
|
||||
respond "This page is for members of the 'media-admins' group." 403
|
||||
}
|
||||
# VictoriaLogs runs with -http.pathPrefix=/logs and emits /logs/... URLs itself, so the prefix is
|
||||
# passed through unchanged. Stripping it would break the UI's own asset paths.
|
||||
redir /logs /logs/select/vmui/ 302
|
||||
redir /logs/ /logs/select/vmui/ 302
|
||||
handle /logs* {
|
||||
reverse_proxy 127.0.0.1:9428
|
||||
}
|
||||
handle /metrics-db* {
|
||||
reverse_proxy 127.0.0.1:8428 # VictoriaMetrics UI + query API
|
||||
}
|
||||
handle /alerts* {
|
||||
reverse_proxy 127.0.0.1:9093 # Alertmanager: what is firing, silences
|
||||
}
|
||||
handle /rules-metrics* {
|
||||
reverse_proxy 127.0.0.1:8880 # vmalert over VictoriaMetrics
|
||||
}
|
||||
handle /rules-logs* {
|
||||
reverse_proxy 127.0.0.1:8881 # vmalert over VictoriaLogs
|
||||
}
|
||||
handle {
|
||||
respond "Vanguard diagnostics
|
||||
|
||||
/logs logs (VictoriaLogs)
|
||||
/metrics-db metrics (VictoriaMetrics)
|
||||
/alerts what is firing (Alertmanager)
|
||||
/rules-metrics metric rules (vmalert)
|
||||
/rules-logs log rules (vmalert)
|
||||
|
||||
Alerts go to Discord. The deadman runs on the host and watches this stack." 200
|
||||
}
|
||||
}
|
||||
|
||||
# vanguard:live
|
||||
live.example.com {
|
||||
encode zstd gzip
|
||||
|
||||
Reference in New Issue
Block a user