Propagate this build's lessons into the scripts and every runbook

addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.

HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.

LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

FRESH-INSTALL.md

@@ -5,7 +5,7 @@ Everything in order. Scripted steps are commands; manual steps are click lists.
 README.md (section numbers given).
 
 ## 0. Before touching the server
-- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `auth`, `watch`, `git`, `bot` → `HOST`.
+- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `diag`, `auth`, `watch`, `git`, `bot` → `HOST`.
   Optional: the bare domain `A` → server IP (it redirects to `https://HOST/`).
 - The Discord bot lives in its **own** repo (ours `VanguardOrg/VanguardBot`), not in this one. Nothing here
   needs it to exist; section 7a wires it up afterwards.
@@ -41,7 +41,7 @@ build, **firewall** (keep this session open; test a new ssh in a second terminal
 
     cd ~/vanguard-stream && ./configure.py --full
 
-Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
+Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`diag`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
 redirects; Discord bot token + server ID (optional now, needed for the plugin feed); "is live" webhook (optional).
 Generates `.env` with every secret, the configs, starts all containers, waits, then runs the post-start fixups
 (Syncthing GUI auth off, Gitea admin user, Jellyfin plugin). `./configure.py --show` prints every credential
@@ -117,6 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro
 6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
    container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
 
+## 7b. Diagnostics — logs, metrics, alerts (README §5h)
+1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector,
+   VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE`
+   behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
+2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and
+   show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting.
+3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing
+   that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`.
+4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D).
+
 ## 8. Pipeline (README "Deployment")
 1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
    → `./switchToClone.sh clone` (keeps `.env`, generated configs, plugin zips) → `./configure.py --apply`.
@@ -341,8 +351,21 @@ L7. dev$ ssh debian@HOST 'docker logs discord-bot --tail 20'
 L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. Needs the `bot-users` group (admins
     need it too, or set BOT_REQUIRE_GROUP= and let the bot's WEB_ALLOWED_GROUPS decide).
 
-## M. Afterwards
+## M. Diagnostics (browser at https://diag.BASE/)
 
-M1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
-M2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
-M3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.
+M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"'
+    → five Up. Vector restarting means a config error: docker logs vector --tail 20.
+M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the
+    stream fields and you should see several distinct ones.
+M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat
+    the deadman watches.
+M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+    → no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it.
+M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that
+    regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty.
+
+## N. Afterwards
+
+N1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
+N2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
+N3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.