Brandyn / Techy
d3ff452546
Deploying before the webhook exists is deliberate, so it must not depend on Alertmanager tolerating an empty webhook_url_file — which I could not verify. configure.py now generates the config and omits the Discord notifier entirely when ALERT_WEBHOOK_URL is unset: the receiver exists, drops notifications, and alerts still evaluate, group and show at /alerts. Setting the variable and re-applying turns delivery on. The URL stays in /etc/vanguard/alert-webhook rather than the config, because that file can be 0640 root:65534 for the container's user while the repo copy cannot. alertmanager.example.yml is the reference copy and is gitignored in generated form, like the other configs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
55 lines
2.3 KiB
YAML
55 lines
2.3 KiB
YAML
# REFERENCE COPY ONLY — configure.py generates observability/alertmanager.yml from ALERT_WEBHOOK_URL and that
|
|
# generated file is what runs (it omits the Discord receiver's notifier entirely when no webhook is set, so
|
|
# the config stays valid and alerts simply are not delivered). Never scp this over the generated one.
|
|
#
|
|
# Alertmanager: groups what vmalert fires and posts it to Discord. The webhook URL is NOT in here — it is a
|
|
# secret, so configure.py writes it to /etc/vanguard/alert-webhook with mode 0600 and this reads it from there.
|
|
#
|
|
# The "heartbeat" route is the other half of the deadman. The Watchdog alert in alerts-metrics.yml always fires,
|
|
# so Alertmanager posts a heartbeat to a file-watching receiver every interval; bin/vanguard-deadman.sh on the
|
|
# host notices when that stops and tells Discord itself. Nothing inside the stack can report its own death.
|
|
|
|
global:
|
|
resolve_timeout: 5m
|
|
|
|
route:
|
|
receiver: discord
|
|
group_by: [alertname, service]
|
|
group_wait: 30s
|
|
group_interval: 5m
|
|
repeat_interval: 12h
|
|
routes:
|
|
- matchers: [severity="none"]
|
|
receiver: heartbeat
|
|
group_wait: 0s
|
|
group_interval: 1m
|
|
repeat_interval: 1m
|
|
- matchers: [severity="critical"]
|
|
receiver: discord
|
|
group_wait: 10s
|
|
repeat_interval: 1h
|
|
|
|
inhibit_rules:
|
|
# While the disk is critical, do not also shout that it is merely filling.
|
|
- source_matchers: [alertname="MediaDiskCritical"]
|
|
target_matchers: [alertname="MediaDiskFilling"]
|
|
# If a core container is gone, its restart-loop alert is noise.
|
|
- source_matchers: [alertname="ContainerGone"]
|
|
target_matchers: [alertname="ContainerRestartLoop"]
|
|
|
|
receivers:
|
|
- name: discord
|
|
discord_configs:
|
|
- webhook_url_file: /etc/vanguard/alert-webhook
|
|
title: '{{ if eq .Status "firing" }}🔴{{ else }}✅{{ end }} {{ .CommonLabels.alertname }}'
|
|
message: |
|
|
{{ range .Alerts }}{{ .Annotations.summary }}
|
|
{{ if .Annotations.detail }}{{ .Annotations.detail }}
|
|
{{ end }}{{ end }}
|
|
- name: heartbeat
|
|
# Goes to dumpd on the host (bridge-only port), which touches a file. bin/vanguard-deadman.sh checks that
|
|
# file is fresh, so a heartbeat never reaches Discord but its ABSENCE does.
|
|
webhook_configs:
|
|
- url: http://127.0.0.1:9095/heartbeat
|
|
send_resolved: false
|