diag host: real path prefixes for every UI, and an index page served as a file

Each service now runs behind the prefix it is published under (-http.pathPrefix, or route-prefix plus
external-url for Alertmanager). Without that, three of the four UIs would have loaded at their sub-path and
then fetched their assets from the domain root, so the pages would have come up broken. The prefixes cascade,
so the scrape paths, vmalert's notifier and remote-write URLs, and the deadman's health probes all follow.

The index is diag/index.html mounted into Caddy, not an inline respond: Caddy reads {...} as placeholders and
the page's CSS is full of braces, so inline would have been mangled. It also matches how the gateway, dock and
clips pages are already served.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bin/vanguard-deadman.sh

@@ -35,11 +35,13 @@ alive() {   # alive <label> <url>
   [[ $code == 2* || $code == 3* ]] || note "$1 is not answering (HTTP ${code:-no response})"
 }
 
+# Each service runs behind the path prefix it is published under on diag.HOST, so its health endpoint
+# lives under that prefix too.
 alive "VictoriaLogs"      "http://127.0.0.1:9428/logs/health"
-alive "VictoriaMetrics" "http://127.0.0.1:8428/health"
+alive "VictoriaMetrics"   "http://127.0.0.1:8428/metrics-db/health"
-alive "vmalert (metrics)" "http://127.0.0.1:8880/health"
+alive "vmalert (metrics)" "http://127.0.0.1:8880/rules-metrics/health"
-alive "vmalert (logs)"    "http://127.0.0.1:8881/health"
+alive "vmalert (logs)"    "http://127.0.0.1:8881/rules-logs/health"
-alive "Alertmanager"    "http://127.0.0.1:9093/-/healthy"
+alive "Alertmanager"      "http://127.0.0.1:9093/alerts/-/healthy"
 
 # 2. the heartbeat. A missing file right after a restart is not yet a fault, so only age counts.
 if [[ -f $HEARTBEAT ]]; then

configure.py

@@ -249,15 +249,9 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 		reverse_proxy 127.0.0.1:8881          # vmalert over VictoriaLogs
 	}}
 	handle {{
-		respond "Vanguard diagnostics
+		root * /srv/diag
-
+		try_files {{path}} /index.html
-/logs            logs (VictoriaLogs)
+		file_server                           # diag/index.html: what each path above is
-/metrics-db      metrics (VictoriaMetrics)
-/alerts          what is firing (Alertmanager)
-/rules-metrics   metric rules (vmalert)
-/rules-logs      log rules (vmalert)
-
-Alerts go to Discord. The deadman runs on the host and watches this stack." 200
 	}}
 }}
 """

diag/index.html

@@ -0,0 +1,37 @@
+<!doctype html>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Vanguard diagnostics</title>
+<style>
+:root{--bg:#121417;--panel:#1c2026;--line:#2b313a;--fg:#f3f1ea;--mut:#a3acb8;--acc:#e5484d}
+*{box-sizing:border-box}
+body{background:var(--bg);color:var(--fg);font:16px/1.55 system-ui,-apple-system,"Segoe UI",sans-serif;margin:0;padding:48px 20px}
+main{max-width:680px;margin:0 auto}
+h1{font-size:30px;margin:0 0 6px;letter-spacing:.3px}
+.lead{color:var(--mut);margin:0 0 30px}
+a.card{display:block;padding:15px 18px;margin:0 0 10px;background:var(--panel);border:1px solid var(--line);
+  border-radius:12px;color:var(--fg);text-decoration:none}
+a.card:hover{border-color:var(--acc)}
+a.card b{display:block;font-size:17px}
+a.card span{color:var(--mut);font-size:14px}
+.note{margin-top:30px;padding:16px 18px;background:var(--panel);border:1px solid var(--line);border-radius:12px;
+  color:var(--mut);font-size:14px}
+.note b{color:var(--fg)}
+footer{margin-top:26px;color:#6b7280;font-size:13px}
+footer a{color:#6b7280}
+</style></head>
+<body><main>
+<h1>Vanguard diagnostics</h1>
+<p class="lead">Logs, metrics and alerts for this server. Visible to <code>media-admins</code>.</p>
+
+<a class="card" href="/logs"><b>Logs</b><span>every container and our own timers, last 30 days · VictoriaLogs</span></a>
+<a class="card" href="/metrics-db"><b>Metrics</b><span>host, containers, MediaMTX, Caddy, last 90 days · VictoriaMetrics</span></a>
+<a class="card" href="/alerts"><b>Alerts</b><span>what is firing now, and silences · Alertmanager</span></a>
+<a class="card" href="/rules-metrics"><b>Metric rules</b><span>disk, load, memory, RAID, container restart loops</span></a>
+<a class="card" href="/rules-logs"><b>Log rules</b><span>sweep failures, publish denials, failed bot releases</span></a>
+
+<div class="note"><b>A deadman runs on the host, outside all of this.</b> Every five minutes it checks that each
+service here still answers, that the alerting path is still delivering, and that the disks are not filling, and
+it reports to Discord itself. Nothing on this page can tell you it has died, which is the point.</div>
+
+<footer>Vanguard Media · <a href="/logout">sign out</a></footer>
+</main></body></html>

docker-compose.yml

@@ -72,6 +72,7 @@ services:
       - ./dock:/srv/dock:ro
       - ./clips:/srv/clips:ro               # clip-intake self-service page
       - ./about:/srv/about:ro               # gateway page (/about)
+      - ./diag:/srv/diag:ro                 # diagnostics index (diag.HOST)
       - /srv/live:/srv/live:ro              # live portal static build (deployLive.sh)
       - caddy_data:/data
       - caddy_config:/config
@@ -395,6 +396,7 @@ services:
       - -storageDataPath=/data
       - -retentionPeriod=90d
       - -httpListenAddr=:8428
+      - -http.pathPrefix=/metrics-db   # served at diag.HOST/metrics-db, so its own UI must emit that prefix
       - -promscrape.config=/etc/vm/scrape.yml
       - -promscrape.config.strictParse=false
     ports:
@@ -412,11 +414,12 @@ services:
     image: docker.io/victoriametrics/vmalert:latest
     container_name: vmalert-metrics
     command:
-      - -datasource.url=http://victoriametrics:8428
+      - -datasource.url=http://victoriametrics:8428/metrics-db
-      - -notifier.url=http://alertmanager:9093
+      - -notifier.url=http://alertmanager:9093/alerts
-      - -remoteWrite.url=http://victoriametrics:8428
+      - -remoteWrite.url=http://victoriametrics:8428/metrics-db
       - -rule=/etc/alerts/alerts-metrics.yml
       - -httpListenAddr=:8880
+      - -http.pathPrefix=/rules-metrics
       - -evaluationInterval=1m
     ports:
       - "127.0.0.1:8880:8880"
@@ -436,10 +439,11 @@ services:
     # LogsQL stats endpoints, which answer in the Prometheus format vmalert expects.
     command:
       - -datasource.url=http://victorialogs:9428/logs
-      - -notifier.url=http://alertmanager:9093
+      - -notifier.url=http://alertmanager:9093/alerts
-      - -remoteWrite.url=http://victoriametrics:8428
+      - -remoteWrite.url=http://victoriametrics:8428/metrics-db
       - -rule=/etc/alerts/alerts-logs.yml
       - -httpListenAddr=:8881
+      - -http.pathPrefix=/rules-logs
       - -evaluationInterval=2m
     ports:
       - "127.0.0.1:8881:8881"
@@ -459,6 +463,8 @@ services:
       - --config.file=/etc/alertmanager/alertmanager.yml
       - --storage.path=/data
       - --web.listen-address=:9093
+      - --web.route-prefix=/alerts
+      - --web.external-url=https://${DIAG_HOST:-diag.example.com}/alerts
     ports:
       - "127.0.0.1:9093:9093"
     volumes:

observability/scrape.yml

@@ -18,5 +18,13 @@ scrape_configs:
     metrics_path: /logs/metrics    # it runs behind -http.pathPrefix=/logs
   - job_name: victoriametrics
     static_configs: [{targets: ["127.0.0.1:8428"]}]
+    metrics_path: /metrics-db/metrics
   - job_name: alertmanager
     static_configs: [{targets: ["alertmanager:9093"]}]
+    metrics_path: /alerts/metrics
+  - job_name: vmalert
+    static_configs: [{targets: ["vmalert-metrics:8880"], labels: {rules: metrics}}]
+    metrics_path: /rules-metrics/metrics
+  - job_name: vmalert-logs
+    static_configs: [{targets: ["vmalert-logs:8881"], labels: {rules: logs}}]
+    metrics_path: /rules-logs/metrics