Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

STATUS.md

@@ -1,6 +1,6 @@
 # Vanguard stack — what runs, what people do, what's unfinished
 
-Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01.
+Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
 Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
 Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).
 
@@ -59,8 +59,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
-| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
-| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
+| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
+| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
+| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
 | Jellyfin | watch page, SSO via Authentik, direct play only | live |
@@ -91,6 +94,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
    (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
    from the bot itself now; an Authentik username is matched to one Discord member by name.
+9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
+9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
 10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
 11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
 
@@ -111,3 +116,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
+- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
+- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
+- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
+- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.