Elastic security, our own metrics, and an alert pipeline that can actually reach Discord
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -72,7 +72,6 @@ services:
|
||||
- ./dock:/srv/dock:ro
|
||||
- ./clips:/srv/clips:ro # clip-intake self-service page
|
||||
- ./about:/srv/about:ro # gateway page (/about)
|
||||
- ./diag:/srv/diag:ro # diagnostics index (diag.HOST)
|
||||
- /srv/live:/srv/live:ro # live portal static build (deployLive.sh)
|
||||
- caddy_data:/data
|
||||
- caddy_config:/config
|
||||
@@ -235,6 +234,17 @@ services:
|
||||
- ${BOT_APP_DIR:-/srv/bot/app}:/app # CI bundle installed by ./deployBotBundle.sh (code + vendor/)
|
||||
- /srv/media/clips:/media/clips # rw: for a clip-submissions feature (writes into clips/<user>/)
|
||||
- /srv/appdata/discord-bot:/data # persistent scratch (sqlite, caches) if the bot wants files
|
||||
# Asks the BOT about itself rather than just proving the port accepts a connection: /health answers 503
|
||||
# unless the Discord gateway is connected AND the database responds, and -f turns that into an unhealthy
|
||||
# container, which Metricbeat's docker healthcheck metricset turns into an alarm. Requires the bot release
|
||||
# that made /health unauthenticated and status-bearing — on older code every check 401s and -f fails,
|
||||
# so ship the bot first. start_period covers command sync and the first roster load.
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS -m 5 http://127.0.0.1:8000/health >/dev/null || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
networks:
|
||||
@@ -330,7 +340,18 @@ services:
|
||||
# machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics.
|
||||
ES_JAVA_OPTS: -Xms2g -Xmx2g
|
||||
discovery.type: single-node
|
||||
xpack.security.enabled: "false" # loopback only, behind Caddy + Authentik; no second login
|
||||
# ON. Kibana's Observability apps (Infrastructure/Metrics) ask Elasticsearch whether the current user
|
||||
# holds a privilege; with security off that API does not exist and they fail on every view. Humans still
|
||||
# never see a password: Kibana signs everyone in as one user (observability/kibana.yml). Caddy +
|
||||
# Authentik + DIAG_REQUIRE_GROUP remains the only real gate.
|
||||
xpack.security.enabled: "true"
|
||||
xpack.security.http.ssl.enabled: "false" # plain HTTP, published on loopback only
|
||||
xpack.security.transport.ssl.enabled: "false" # single-node: no transport traffic, and the check is skipped
|
||||
xpack.security.enrollment.enabled: "false"
|
||||
xpack.security.autoconfiguration.enabled: "false"
|
||||
# Bootstraps the elastic superuser on first start with security on. configure.py generates it and then
|
||||
# sets up kibana_system and the service users; the error below means .env predates that.
|
||||
ELASTIC_PASSWORD: "${ES_ELASTIC_PASSWORD:?run ./configure.py --apply first — it generates the Elastic passwords}"
|
||||
xpack.ml.enabled: "false" # not used, and it reserves memory
|
||||
cluster.routing.allocation.disk.watermark.low: 85%
|
||||
cluster.routing.allocation.disk.watermark.high: 90%
|
||||
@@ -344,7 +365,7 @@ services:
|
||||
volumes:
|
||||
- /srv/appdata/elasticsearch:/usr/share/elasticsearch/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
|
||||
test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
|
||||
interval: 20s
|
||||
timeout: 10s
|
||||
retries: 12
|
||||
@@ -356,14 +377,16 @@ services:
|
||||
labels: { vanguard.service: kibana, vanguard.category: infra }
|
||||
image: docker.elastic.co/kibana/kibana:9.2.0
|
||||
container_name: kibana
|
||||
# Everything non-secret lives in observability/kibana.yml (which explains the auth model); only the
|
||||
# generated secrets arrive as environment variables, which that file interpolates.
|
||||
environment:
|
||||
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
|
||||
SERVER_PUBLICBASEURL: https://${DIAG_HOST:-diag.example.com}
|
||||
SERVER_BASEPATH: "" # served at the root of diag.HOST, so no prefix rewriting to get wrong
|
||||
TELEMETRY_OPTIN: "false"
|
||||
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
XPACK_REPORTING_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
DIAG_HOST: ${DIAG_HOST:-diag.example.com}
|
||||
KIBANA_ENCRYPTION_KEY: ${KIBANA_ENCRYPTION_KEY}
|
||||
ES_KIBANA_PASSWORD: "${ES_KIBANA_PASSWORD:?run ./configure.py --apply first}"
|
||||
VG_DIAG_USER: ${VG_DIAG_USER:-vanguard_diag}
|
||||
VG_DIAG_PASSWORD: "${VG_DIAG_PASSWORD:?run ./configure.py --apply first}"
|
||||
volumes:
|
||||
- ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
|
||||
ports:
|
||||
- "127.0.0.1:5601:5601"
|
||||
depends_on:
|
||||
@@ -375,6 +398,9 @@ services:
|
||||
labels: { vanguard.service: filebeat, vanguard.category: infra }
|
||||
image: docker.elastic.co/beats/filebeat:9.2.0
|
||||
container_name: filebeat
|
||||
environment: # Elasticsearch requires auth now; the config interpolates these
|
||||
ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
|
||||
ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
|
||||
user: root
|
||||
command: ["filebeat", "-e", "--strict.perms=false"]
|
||||
volumes:
|
||||
@@ -382,6 +408,7 @@ services:
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- /var/run/docker.sock:/var/run/docker.sock # autodiscover reads container names and our vanguard.* labels
|
||||
- /var/log/journal:/var/log/journal:ro # our own systemd units
|
||||
- /run/log/journal:/run/log/journal:ro # ...and the volatile journal, if this host has no persistent one
|
||||
- /etc/machine-id:/etc/machine-id:ro
|
||||
- filebeat_data:/usr/share/filebeat/data
|
||||
depends_on:
|
||||
@@ -393,6 +420,9 @@ services:
|
||||
labels: { vanguard.service: metricbeat, vanguard.category: infra }
|
||||
image: docker.elastic.co/beats/metricbeat:9.2.0
|
||||
container_name: metricbeat
|
||||
environment: # Elasticsearch requires auth now; the config interpolates these
|
||||
ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
|
||||
ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
|
||||
user: root
|
||||
command: ["metricbeat", "-e", "--strict.perms=false"]
|
||||
# system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards.
|
||||
|
||||
Reference in New Issue
Block a user