Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

observability/kibana.yml

@@ -0,0 +1,41 @@
+# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
+# into .env, so this file is committed and contains none.
+#
+# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
+# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
+# whether the current user holds a privilege. With security off that API does not exist, and the
+# Infrastructure / Metrics app fails on every page view with
+#   no handler found for uri [/_security/user/_has_privileges]
+# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
+# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
+# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
+# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
+#
+# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
+# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
+# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
+# scheme and have Caddy inject a key chosen by Authentik group).
+
+server.host: "0.0.0.0"
+server.publicBaseUrl: "https://${DIAG_HOST}"
+# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
+# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
+# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
+telemetry.optIn: false
+
+elasticsearch.hosts: ["http://elasticsearch:9200"]
+elasticsearch.username: "kibana_system"     # Kibana's own internal account, NOT what visitors become
+elasticsearch.password: "${ES_KIBANA_PASSWORD}"
+
+# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
+xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+
+# The only provider, so there is no login page and no login selector: every request is already signed in.
+xpack.security.authc.providers:
+  anonymous.anonymous1:
+    order: 0
+    credentials:
+      username: "${VG_DIAG_USER}"
+      password: "${VG_DIAG_PASSWORD}"