Commit Graph

5 Commits

Author SHA1 Message Date
a77051c496 apply creates the observability data dirs; Caddyfile.example regenerated with the diag block
Without those three directories Docker creates them root-owned and victorialogs, victoriametrics (uid 1000)
and alertmanager (65534) cannot write, so the apply has to make them the way it already makes /srv/bot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 16:18:13 -06:00
8ae43905dc Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host:
  victorialogs      30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged
  vector            docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path)
  victoriametrics   90d, scrapes the four taps that were already live plus the new services
  vmalert-metrics   11 metric rules: disk, load, memory, RAID, restart loops, a core container gone
  vmalert-logs      5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble
  alertmanager      groups and inhibits, posts to Discord, webhook read from a 0600 file not the config

The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above:
curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through
vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once.
Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once.

Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host
and is recorded as such in vector.yaml with the two upgrade paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:45:58 -06:00
855651c5a4 Metrics taps: node_exporter + cAdvisor containers, MediaMTX and Caddy metrics on loopback (Prometheus format, store-agnostic)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 19:55:42 -06:00
18f4e07de8 Split hosts: media (gateway at /), files, clips, sync; live gains /dock, /dock/clock, /dump, /me, /time
configure.py: three new host roles (files/sync/clips) in --full, --hosts, .env, Caddy generator;
main host serves about/ at the root (/about* redirects); /time only where it is fetched same-origin.
about/index.html links across hosts; addCommunityMediaServer.sh asks for the new names; Caddyfile.example
regenerated; README/FRESH-INSTALL/HOSTNAME-CHANGE (procedure C)/STATUS updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-01 01:38:35 -05:00
b2869c3740 Generated configs out of the repo: *.example.* templates only; configure.py refuses to apply over CHANGEME placeholders
A deploy copied mediamtx.yml (template) over the server's generated file and the next apply
wiped all streamers. Generated files are now gitignored so scp -r cannot clobber them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-31 00:32:30 -05:00