Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
50
observability/alertmanager.yml
Normal file
50
observability/alertmanager.yml
Normal file
@@ -0,0 +1,50 @@
|
||||
# Alertmanager: groups what vmalert fires and posts it to Discord. The webhook URL is NOT in here — it is a
|
||||
# secret, so configure.py writes it to /etc/vanguard/alert-webhook with mode 0600 and this reads it from there.
|
||||
#
|
||||
# The "heartbeat" route is the other half of the deadman. The Watchdog alert in alerts-metrics.yml always fires,
|
||||
# so Alertmanager posts a heartbeat to a file-watching receiver every interval; bin/vanguard-deadman.sh on the
|
||||
# host notices when that stops and tells Discord itself. Nothing inside the stack can report its own death.
|
||||
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: discord
|
||||
group_by: [alertname, service]
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
routes:
|
||||
- matchers: [severity="none"]
|
||||
receiver: heartbeat
|
||||
group_wait: 0s
|
||||
group_interval: 1m
|
||||
repeat_interval: 1m
|
||||
- matchers: [severity="critical"]
|
||||
receiver: discord
|
||||
group_wait: 10s
|
||||
repeat_interval: 1h
|
||||
|
||||
inhibit_rules:
|
||||
# While the disk is critical, do not also shout that it is merely filling.
|
||||
- source_matchers: [alertname="MediaDiskCritical"]
|
||||
target_matchers: [alertname="MediaDiskFilling"]
|
||||
# If a core container is gone, its restart-loop alert is noise.
|
||||
- source_matchers: [alertname="ContainerGone"]
|
||||
target_matchers: [alertname="ContainerRestartLoop"]
|
||||
|
||||
receivers:
|
||||
- name: discord
|
||||
discord_configs:
|
||||
- webhook_url_file: /etc/vanguard/alert-webhook
|
||||
title: '{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}'
|
||||
message: |
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ if .Annotations.detail }}{{ .Annotations.detail }}
|
||||
{{ end }}{{ end }}
|
||||
- name: heartbeat
|
||||
# Goes to dumpd on the host (bridge-only port), which touches a file. bin/vanguard-deadman.sh checks that
|
||||
# file is fresh, so a heartbeat never reaches Discord but its ABSENCE does.
|
||||
webhook_configs:
|
||||
- url: http://127.0.0.1:9095/heartbeat
|
||||
send_resolved: false
|
||||
Reference in New Issue
Block a user