Brandyn / Techy
88bb28989c
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and Caddy's Prometheus endpoints. Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box whose spare capacity belongs to x264), single-node discovery, security off because it only listens on loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart. diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links. The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
51 lines
1.7 KiB
YAML
51 lines
1.7 KiB
YAML
# Filebeat: container logs + our own systemd units -> Elasticsearch.
|
|
# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
|
|
# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
|
|
# needs no parsing. See LOGGING.md for what each service's lines look like.
|
|
|
|
filebeat.autodiscover:
|
|
providers:
|
|
- type: docker
|
|
hints.enabled: false
|
|
templates:
|
|
- condition: ~ # every container
|
|
config:
|
|
- type: container
|
|
paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
|
|
|
|
filebeat.inputs:
|
|
- type: journald
|
|
id: vanguard-units
|
|
include_matches:
|
|
match:
|
|
- "_SYSTEMD_UNIT=dumpd.service"
|
|
- "_SYSTEMD_UNIT=normalize-sweep.service"
|
|
- "_SYSTEMD_UNIT=trash-purge.service"
|
|
- "_SYSTEMD_UNIT=vgdeband.service"
|
|
- "_SYSTEMD_UNIT=vanguard-deadman.service"
|
|
|
|
processors:
|
|
- add_docker_metadata: ~
|
|
- add_host_metadata: ~
|
|
# promote our labels to real fields so Kibana filters on service.name and vanguard.category
|
|
- copy_fields:
|
|
fields:
|
|
- from: container.labels.vanguard_service
|
|
to: service.name
|
|
- from: container.labels.vanguard_category
|
|
to: vanguard.category
|
|
fail_on_error: false
|
|
ignore_missing: true
|
|
- drop_fields:
|
|
fields: ["agent.ephemeral_id", "ecs.version", "input.type", "log.offset"]
|
|
ignore_missing: true
|
|
|
|
output.elasticsearch:
|
|
hosts: ["http://elasticsearch:9200"]
|
|
|
|
setup.kibana:
|
|
host: "http://kibana:5601"
|
|
setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h)
|
|
|
|
logging.level: info
|