Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
22
observability/scrape.yml
Normal file
22
observability/scrape.yml
Normal file
@@ -0,0 +1,22 @@
|
||||
# What VictoriaMetrics scrapes. Every target is loopback-only on this host (LOGGING.md "Metrics taps").
|
||||
global:
|
||||
scrape_interval: 30s
|
||||
external_labels:
|
||||
host: vanguard
|
||||
|
||||
scrape_configs:
|
||||
- job_name: host # CPU, RAM, disk space + I/O, network, load, md RAID
|
||||
static_configs: [{targets: ["host.docker.internal:9100"]}]
|
||||
- job_name: containers # per container CPU/memory/network/block IO, labelled vanguard.*
|
||||
static_configs: [{targets: ["host.docker.internal:9101"]}]
|
||||
- job_name: mediamtx # per path bytes/s, readers, publishers, dropped packets
|
||||
static_configs: [{targets: ["host.docker.internal:9998"]}]
|
||||
- job_name: caddy # requests by host and status, durations, TLS
|
||||
static_configs: [{targets: ["host.docker.internal:2019"]}]
|
||||
- job_name: victorialogs
|
||||
static_configs: [{targets: ["victorialogs:9428"]}]
|
||||
metrics_path: /logs/metrics # it runs behind -http.pathPrefix=/logs
|
||||
- job_name: victoriametrics
|
||||
static_configs: [{targets: ["127.0.0.1:8428"]}]
|
||||
- job_name: alertmanager
|
||||
static_configs: [{targets: ["alertmanager:9093"]}]
|
||||
Reference in New Issue
Block a user