Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
90 lines
3.9 KiB
YAML
90 lines
3.9 KiB
YAML
# Filebeat: container logs + our own systemd units -> Elasticsearch.
|
|
# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
|
|
# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
|
|
# needs no parsing. See LOGGING.md for what each service's lines look like.
|
|
#
|
|
# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and
|
|
# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the
|
|
# process up with zero harvesters. Container logs are read by filestream + the container parser, and every
|
|
# filestream input needs its own stable id.
|
|
|
|
filebeat.autodiscover:
|
|
providers:
|
|
- type: docker
|
|
hints.enabled: false
|
|
templates:
|
|
- condition: ~ # every container
|
|
config:
|
|
- type: filestream
|
|
id: container-${data.docker.container.id}
|
|
paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
|
|
parsers:
|
|
- container: ~
|
|
prospector.scanner.symlinks: true
|
|
# Stamp the id here rather than letting add_docker_metadata guess it from the path.
|
|
processors:
|
|
- add_fields:
|
|
target: container
|
|
fields:
|
|
id: "${data.docker.container.id}"
|
|
name: "${data.docker.container.name}"
|
|
image.name: "${data.docker.container.image}"
|
|
|
|
filebeat.inputs:
|
|
- type: journald
|
|
id: vanguard-units
|
|
include_matches:
|
|
match:
|
|
- "_SYSTEMD_UNIT=dumpd.service"
|
|
- "_SYSTEMD_UNIT=normalize-sweep.service"
|
|
- "_SYSTEMD_UNIT=trash-purge.service"
|
|
- "_SYSTEMD_UNIT=vgdeband.service"
|
|
- "_SYSTEMD_UNIT=vanguard-deadman.service"
|
|
|
|
processors:
|
|
- add_docker_metadata:
|
|
match_fields: ["container.id"] # the id the autodiscover template stamped above
|
|
- add_host_metadata: ~
|
|
# promote our labels to real fields so Kibana filters on service.name and vanguard.category
|
|
- copy_fields:
|
|
fields:
|
|
- from: container.labels.vanguard_service
|
|
to: service.name
|
|
- from: container.labels.vanguard_category
|
|
to: vanguard.category
|
|
fail_on_error: false
|
|
ignore_missing: true
|
|
# The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in
|
|
# a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's
|
|
# vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target
|
|
# that already exists, so the first rule that matches wins and the other is skipped.
|
|
- copy_fields:
|
|
fields:
|
|
- from: container.labels.vanguard_service
|
|
to: event.dataset
|
|
- from: systemd.unit
|
|
to: event.dataset
|
|
fail_on_error: false
|
|
ignore_missing: true
|
|
- drop_fields:
|
|
fields: ["agent.ephemeral_id", "ecs.version", "log.offset"]
|
|
ignore_missing: true
|
|
|
|
output.elasticsearch:
|
|
hosts: ["http://elasticsearch:9200"]
|
|
# Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
|
|
# come from the container environment, which compose fills from .env — no secret is committed here.
|
|
username: "${ES_BEATS_USER}"
|
|
password: "${ES_BEATS_PASSWORD}"
|
|
|
|
setup.kibana: # no credentials: Kibana's anonymous provider signs these in
|
|
host: "http://kibana:5601"
|
|
setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h)
|
|
# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
|
|
# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
|
|
setup.template.overwrite: true
|
|
setup.template.settings:
|
|
index.number_of_replicas: 0
|
|
|
|
logging.level: info
|