From 6f7fb29ae099cb21b2db9dcb74dd74b00ee60874 Mon Sep 17 00:00:00 2001 From: Brandyn / Techy Date: Tue, 3 Nov 2026 16:55:39 -0600 Subject: [PATCH] metricbeat sets the same correlation fields as filebeat, so one dashboard spans logs and metrics Both Beats already wrote to the same Elasticsearch, but a metric event carried container.name while a log event carried service.name and vanguard.category, so nothing joined them and a dashboard could only show the two side by side. Metricbeat now runs add_docker_metadata and the same copy_fields, giving both sides @timestamp, host.name, container.name, service.name and vanguard.category. LOGGING.md documents the join keys and the one Kibana step that makes it work: a single data view over filebeat-*,metricbeat-* so one filter applies to metric panels and log panels together. Co-Authored-By: Claude Opus 5 (1M context) --- LOGGING.md | 32 ++++++++++++++++++++++++++++++++ observability/metricbeat.yml | 11 +++++++++++ 2 files changed, 43 insertions(+) diff --git a/LOGGING.md b/LOGGING.md index c4a3d92..d5387c1 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -86,6 +86,38 @@ One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 da ## As built (Phase 13) +| container | port | job | +|---|---|---| +| `elasticsearch` | 9200 | one store for **both** logs and metrics. 2 GB heap, single node, security off (loopback only) | +| `kibana` | 5601 | the single pane: Discover for logs, dashboards, alerting. Served at the root of `diag.BASE` | +| `filebeat` | — | container logs (autodiscover) + our journal units | +| `metricbeat` | — | host + Docker metrics (retires node_exporter and cAdvisor) + MediaMTX and Caddy Prometheus endpoints | + +**Logs and metrics correlate because both Beats write the same fields into the same store.** That is what makes +one dashboard able to show a CPU spike next to the errors that caused it, rather than two dashboards side by +side: + +| field | set by | use | +|---|---|---| +| `@timestamp` | both (ECS) | every panel shares the dashboard's time range | +| `host.name` | both | one box today, but the field is there | +| `container.name` | both | per-service metrics joined to that service's log lines | +| `service.name` | both, from the `vanguard.service` label | the name you actually think in | +| `vanguard.category` | both, from the label | filter a whole area at once: stream, media, auth, intake, deploy | + +To get a dashboard that filters across both, make **one data view** spanning `filebeat-*,metricbeat-*` in +Kibana (Stack Management → Data Views). A single filter such as `service.name: mediamtx` then applies to the +metric panels and the log panel together. Metricbeat also ships its own system and Docker dashboards, which are +a good starting point to copy panels from. + +Retention is one edit in Kibana rather than config: Stack Management → Index Lifecycle Policies → the beats +policy → add a delete phase. Doing it there rather than in a file keeps it adjustable without a redeploy. + +### Old notes + + +(superseded 2026-11-03 — the Victoria/vmalert/Alertmanager shape, kept for the reasoning) + | container | port | job | |---|---|---| | `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged | diff --git a/observability/metricbeat.yml b/observability/metricbeat.yml index 7e7c4f4..e87eef5 100644 --- a/observability/metricbeat.yml +++ b/observability/metricbeat.yml @@ -29,6 +29,17 @@ metricbeat.modules: processors: - add_host_metadata: ~ + # The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a + # metric event carries container.name while a log event carries service.name, and nothing joins them. + - add_docker_metadata: ~ + - copy_fields: + fields: + - from: container.labels.vanguard_service + to: service.name + - from: container.labels.vanguard_category + to: vanguard.category + fail_on_error: false + ignore_missing: true output.elasticsearch: hosts: ["http://127.0.0.1:9200"]