Brandyn / Techy
88bb28989c
Eight observability containers become four. Elasticsearch stores logs and metrics, Kibana is the one UI for Discover, dashboards and alerting, Filebeat ships container logs plus our journal units, and Metricbeat covers host and Docker metrics — which retires node_exporter and cAdvisor too — while still scraping MediaMTX's and Caddy's Prometheus endpoints. Pinned rather than left to chance: a fixed 2 GB heap (a self-sizing JVM would claim a quarter of a 62 GB box whose spare capacity belongs to x264), single-node discovery, security off because it only listens on loopback behind the Authentik gate, ML off, and vm.max_map_count set and persisted since ES will not start without it. Kibana gets a fixed encryption key or it loses its saved objects on every restart. diag.HOST is now just Kibana at the root: no path prefixes to get wrong and no index page of links. The deadman is the only thing kept from the old stack, repointed at Elastic. It checks that ES answers and is not red, that no index has been flipped read-only by the disk watermark (which does not clear itself when space is freed, so ingestion stays silently dead), that Kibana is available, that each Beat has actually written a document recently rather than merely running, that the Kibana heartbeat still arrives, and the disks. Verified against a mock Elasticsearch for all five failure modes plus the healthy case staying silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
41 lines
1.4 KiB
YAML
41 lines
1.4 KiB
YAML
# Metricbeat: host and per-container metrics, replacing node_exporter and cAdvisor, plus the Prometheus
|
|
# endpoints the stack already exposes. Runs on the host network so the system module sees the real interfaces
|
|
# and so it can reach MediaMTX, Caddy, Elasticsearch and Kibana on loopback.
|
|
|
|
metricbeat.modules:
|
|
- module: system
|
|
period: 30s
|
|
metricsets: [cpu, load, memory, network, process, process_summary, socket_summary, filesystem, fsstat, diskio, uptime]
|
|
processes: ['.*']
|
|
process.include_top_n:
|
|
by_cpu: 10
|
|
by_memory: 10
|
|
system.hostfs: /hostfs # the host's real filesystems, not the container's
|
|
|
|
- module: docker
|
|
period: 30s
|
|
hosts: ["unix:///var/run/docker.sock"]
|
|
metricsets: [container, cpu, diskio, healthcheck, info, memory, network]
|
|
|
|
- module: prometheus # MediaMTX: per path bytes/s, readers, publishers, dropped packets
|
|
period: 30s
|
|
metrics_path: /metrics
|
|
hosts: ["http://127.0.0.1:9998"]
|
|
|
|
- module: prometheus # Caddy: requests by host and status, durations, TLS
|
|
period: 60s
|
|
metrics_path: /metrics
|
|
hosts: ["http://127.0.0.1:2019"]
|
|
|
|
processors:
|
|
- add_host_metadata: ~
|
|
|
|
output.elasticsearch:
|
|
hosts: ["http://127.0.0.1:9200"]
|
|
|
|
setup.kibana:
|
|
host: "http://127.0.0.1:5601"
|
|
setup.dashboards.enabled: true # the system and docker dashboards, so there is something to look at
|
|
|
|
logging.level: info
|