Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42 lines
2.4 KiB
YAML
42 lines
2.4 KiB
YAML
# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
|
|
# into .env, so this file is committed and contains none.
|
|
#
|
|
# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
|
|
# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
|
|
# whether the current user holds a privilege. With security off that API does not exist, and the
|
|
# Infrastructure / Metrics app fails on every page view with
|
|
# no handler found for uri [/_security/user/_has_privileges]
|
|
# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
|
|
# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
|
|
# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
|
|
# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
|
|
#
|
|
# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
|
|
# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
|
|
# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
|
|
# scheme and have Caddy inject a key chosen by Authentik group).
|
|
|
|
server.host: "0.0.0.0"
|
|
server.publicBaseUrl: "https://${DIAG_HOST}"
|
|
# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
|
|
# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
|
|
# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
|
|
telemetry.optIn: false
|
|
|
|
elasticsearch.hosts: ["http://elasticsearch:9200"]
|
|
elasticsearch.username: "kibana_system" # Kibana's own internal account, NOT what visitors become
|
|
elasticsearch.password: "${ES_KIBANA_PASSWORD}"
|
|
|
|
# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
|
|
xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
|
xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
|
xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
|
|
|
# The only provider, so there is no login page and no login selector: every request is already signed in.
|
|
xpack.security.authc.providers:
|
|
anonymous.anonymous1:
|
|
order: 0
|
|
credentials:
|
|
username: "${VG_DIAG_USER}"
|
|
password: "${VG_DIAG_PASSWORD}"
|