Elastic security, our own metrics, and an alert pipeline that can actually reach Discord
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2,6 +2,11 @@
|
||||
# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
|
||||
# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
|
||||
# needs no parsing. See LOGGING.md for what each service's lines look like.
|
||||
#
|
||||
# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and
|
||||
# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the
|
||||
# process up with zero harvesters. Container logs are read by filestream + the container parser, and every
|
||||
# filestream input needs its own stable id.
|
||||
|
||||
filebeat.autodiscover:
|
||||
providers:
|
||||
@@ -10,8 +15,20 @@ filebeat.autodiscover:
|
||||
templates:
|
||||
- condition: ~ # every container
|
||||
config:
|
||||
- type: container
|
||||
- type: filestream
|
||||
id: container-${data.docker.container.id}
|
||||
paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
|
||||
parsers:
|
||||
- container: ~
|
||||
prospector.scanner.symlinks: true
|
||||
# Stamp the id here rather than letting add_docker_metadata guess it from the path.
|
||||
processors:
|
||||
- add_fields:
|
||||
target: container
|
||||
fields:
|
||||
id: "${data.docker.container.id}"
|
||||
name: "${data.docker.container.name}"
|
||||
image.name: "${data.docker.container.image}"
|
||||
|
||||
filebeat.inputs:
|
||||
- type: journald
|
||||
@@ -25,7 +42,8 @@ filebeat.inputs:
|
||||
- "_SYSTEMD_UNIT=vanguard-deadman.service"
|
||||
|
||||
processors:
|
||||
- add_docker_metadata: ~
|
||||
- add_docker_metadata:
|
||||
match_fields: ["container.id"] # the id the autodiscover template stamped above
|
||||
- add_host_metadata: ~
|
||||
# promote our labels to real fields so Kibana filters on service.name and vanguard.category
|
||||
- copy_fields:
|
||||
@@ -36,15 +54,36 @@ processors:
|
||||
to: vanguard.category
|
||||
fail_on_error: false
|
||||
ignore_missing: true
|
||||
# The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in
|
||||
# a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's
|
||||
# vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target
|
||||
# that already exists, so the first rule that matches wins and the other is skipped.
|
||||
- copy_fields:
|
||||
fields:
|
||||
- from: container.labels.vanguard_service
|
||||
to: event.dataset
|
||||
- from: systemd.unit
|
||||
to: event.dataset
|
||||
fail_on_error: false
|
||||
ignore_missing: true
|
||||
- drop_fields:
|
||||
fields: ["agent.ephemeral_id", "ecs.version", "input.type", "log.offset"]
|
||||
fields: ["agent.ephemeral_id", "ecs.version", "log.offset"]
|
||||
ignore_missing: true
|
||||
|
||||
output.elasticsearch:
|
||||
hosts: ["http://elasticsearch:9200"]
|
||||
# Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
|
||||
# come from the container environment, which compose fills from .env — no secret is committed here.
|
||||
username: "${ES_BEATS_USER}"
|
||||
password: "${ES_BEATS_PASSWORD}"
|
||||
|
||||
setup.kibana:
|
||||
setup.kibana: # no credentials: Kibana's anonymous provider signs these in
|
||||
host: "http://kibana:5601"
|
||||
setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h)
|
||||
# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
|
||||
# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
|
||||
setup.template.overwrite: true
|
||||
setup.template.settings:
|
||||
index.number_of_replicas: 0
|
||||
|
||||
logging.level: info
|
||||
|
||||
150
observability/kibana-alarms.py
Executable file
150
observability/kibana-alarms.py
Executable file
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The real alarm rules. Idempotent: creates what is missing, leaves what exists.
|
||||
|
||||
Every rule is an Elasticsearch query rule that counts matching documents in a window and writes one document
|
||||
into the alert bus through the "Vanguard alert index" connector, which bin/vanguard-alerts.sh delivers to
|
||||
Discord. Run observability/kibana-setup.py first — it creates that connector.
|
||||
|
||||
FIRING CADENCE: notify_when=onThrottleInterval with throttle=12h, so an alarm that stays true writes again
|
||||
every 12 hours and stops when the condition clears. The delivery script's own 15-minute per-rule suppression
|
||||
sits well inside that and never interferes.
|
||||
|
||||
WHAT IS DELIBERATELY NOT HERE
|
||||
Elasticsearch, Kibana and Beat liveness, and the disks, are bin/vanguard-deadman.sh's job. Nothing that
|
||||
depends on Kibana being alive can report that Kibana is dead, so those checks stay outside the stack and
|
||||
duplicating them here would be worse than useless: they would go quiet in exactly the case that matters.
|
||||
|
||||
pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
|
||||
KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
|
||||
-H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
|
||||
| python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
|
||||
KEY="$KEY" python3 observability/kibana-alarms.py
|
||||
|
||||
To change a rule, edit it in Kibana or delete it there and re-run this. Thresholds here are the defaults:
|
||||
FREE_PCT 15 alarm when a real filesystem or memory has less than this percent free
|
||||
STUCK_SECONDS 3600 alarm when a finished recording has waited this long to be normalised
|
||||
THROTTLE 12h how often an unresolved alarm repeats
|
||||
"""
|
||||
import json, os, urllib.error, urllib.request
|
||||
|
||||
KIBANA = "http://127.0.0.1:5601"
|
||||
HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
|
||||
"Authorization": "ApiKey " + os.environ["KEY"]}
|
||||
CONNECTOR_NAME = "Vanguard alert index"
|
||||
|
||||
FREE_PCT = float(os.environ.get("FREE_PCT", "15"))
|
||||
STUCK_SECONDS = int(os.environ.get("STUCK_SECONDS", "3600"))
|
||||
THROTTLE = os.environ.get("THROTTLE", "12h")
|
||||
USED_MAX = round(1 - FREE_PCT / 100, 4) # 15% free -> used.pct >= 0.85
|
||||
|
||||
# Filesystems that are not disks. Without this, every tmpfs and overlay mount reports ~100% used and the
|
||||
# disk alarm would fire permanently on something that cannot fill up.
|
||||
PSEUDO_FS = ["tmpfs", "devtmpfs", "overlay", "squashfs", "ramfs", "autofs", "nsfs", "tracefs", "debugfs",
|
||||
"configfs", "mqueue", "hugetlbfs", "pstore", "bpf", "devpts", "proc", "sysfs", "cgroup",
|
||||
"cgroup2", "fusectl", "securityfs", "efivarfs", "binfmt_misc", "fuse.lxcfs", "iso9660"]
|
||||
|
||||
def rule(name, level, why, query, *, index="metricbeat-*", window=10, comparator=">", threshold=0):
|
||||
return {"name": name, "level": level, "why": why, "query": query,
|
||||
"index": index, "window": window, "comparator": comparator, "threshold": threshold}
|
||||
|
||||
ALARMS = [
|
||||
rule("Disk free below %g%%" % FREE_PCT, "error",
|
||||
"A filesystem is nearly full. Elasticsearch turns its indices READ-ONLY at the flood watermark and "
|
||||
"does not undo that by itself, and recordings stop being published.",
|
||||
{"query": {"bool": {
|
||||
"filter": [{"range": {"system.filesystem.used.pct": {"gte": USED_MAX}}}],
|
||||
"must_not": [{"terms": {"system.filesystem.type": PSEUDO_FS}}]}}}),
|
||||
|
||||
rule("Memory free below %g%%" % FREE_PCT, "warn",
|
||||
"Memory is nearly exhausted. Elasticsearch has a pinned 2 GB heap, so the pressure is usually "
|
||||
"ffmpeg or Jellyfin, and the kernel will start killing processes rather than queueing.",
|
||||
{"query": {"range": {"system.memory.actual.used.pct": {"gte": USED_MAX}}}}),
|
||||
|
||||
rule("Discord bot not running", "error",
|
||||
"No metrics from the discord-bot container for 5 minutes: it is stopped, removed, or restarting too "
|
||||
"fast to be sampled. Nothing it does is happening.",
|
||||
{"query": {"term": {"container.name": "discord-bot"}}},
|
||||
window=5, comparator="<", threshold=1),
|
||||
|
||||
rule("Discord bot unhealthy", "error",
|
||||
"The bot is running but reports itself degraded: its /health says the Discord gateway is "
|
||||
"disconnected or the database does not answer. It is up and doing nothing.",
|
||||
{"query": {"bool": {"filter": [{"term": {"container.name": "discord-bot"}},
|
||||
{"term": {"docker.healthcheck.status": "unhealthy"}}]}}},
|
||||
window=5),
|
||||
|
||||
rule("Recordings stuck waiting to be normalised", "warn",
|
||||
"A finished recording has waited over %d minutes in .live/. It is not in Jellyfin, not in copyparty "
|
||||
"and not announced, so to everyone else it does not exist." % (STUCK_SECONDS // 60),
|
||||
{"query": {"range": {"prometheus.vanguard_media_pending_oldest_seconds.value": {"gt": STUCK_SECONDS}}}}),
|
||||
|
||||
rule("Normalise failures parked", "warn",
|
||||
"One or more files have a normalize-failed log, so the sweep skips them forever until it is deleted. "
|
||||
"The log now contains ffmpeg's stderr, which says why.",
|
||||
{"query": {"range": {"prometheus.vanguard_media_failed_files.value": {"gt": 0}}}}),
|
||||
]
|
||||
|
||||
def req(method, path, body=None):
|
||||
request = urllib.request.Request(KIBANA + path, method=method,
|
||||
data=None if body is None else json.dumps(body).encode())
|
||||
for key, value in HEADERS.items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
raw = response.read()
|
||||
return json.loads(raw) if raw else {}
|
||||
except urllib.error.HTTPError as error:
|
||||
print(f" {method} {path} -> HTTP {error.code}")
|
||||
print(" " + error.read().decode(errors="replace")[:600])
|
||||
raise SystemExit(1)
|
||||
|
||||
connectors = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
|
||||
if CONNECTOR_NAME not in connectors:
|
||||
raise SystemExit(f"connector {CONNECTOR_NAME!r} is missing — run observability/kibana-setup.py first")
|
||||
connector_id = connectors[CONNECTOR_NAME]
|
||||
|
||||
present = {r["name"] for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])}
|
||||
created = skipped = 0
|
||||
for alarm in ALARMS:
|
||||
if alarm["name"] in present:
|
||||
print("already present:", alarm["name"])
|
||||
skipped += 1
|
||||
continue
|
||||
made = req("POST", "/api/alerting/rule", {
|
||||
"name": alarm["name"],
|
||||
"rule_type_id": ".es-query",
|
||||
"consumer": "stackAlerts",
|
||||
"enabled": True,
|
||||
"schedule": {"interval": "1m"},
|
||||
"params": {
|
||||
"searchType": "esQuery",
|
||||
"index": [alarm["index"]],
|
||||
"timeField": "@timestamp",
|
||||
"esQuery": json.dumps(alarm["query"]),
|
||||
"size": 1,
|
||||
"aggType": "count",
|
||||
"groupBy": "all",
|
||||
"thresholdComparator": alarm["comparator"],
|
||||
"threshold": [alarm["threshold"]],
|
||||
"timeWindowSize": alarm["window"],
|
||||
"timeWindowUnit": "m",
|
||||
"excludeHitsFromPreviousRun": False,
|
||||
},
|
||||
"actions": [{
|
||||
"group": "query matched",
|
||||
"id": connector_id,
|
||||
"params": {"documents": [{
|
||||
"kind": "alarm",
|
||||
"rule": "{{rule.name}}",
|
||||
"level": alarm["level"],
|
||||
"message": alarm["why"] + " [{{context.message}}]",
|
||||
}]},
|
||||
# every 12 h for as long as it stays true, then silence when it clears
|
||||
"frequency": {"summary": False, "notify_when": "onThrottleInterval", "throttle": THROTTLE},
|
||||
}],
|
||||
})
|
||||
print("created:", made["name"], "|", alarm["level"], "| repeats every", THROTTLE)
|
||||
created += 1
|
||||
|
||||
print(f"\n{created} created, {skipped} already there. Disk/memory threshold {FREE_PCT:g}% free; "
|
||||
f"stuck after {STUCK_SECONDS}s.")
|
||||
161
observability/kibana-setup.py
Executable file
161
observability/kibana-setup.py
Executable file
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Kibana first-run state. Idempotent; safe to re-run. configure.py --apply runs this for you.
|
||||
|
||||
connector "Vanguard alert index" — the .index connector, writing into the vanguard-alerts index
|
||||
rule "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
|
||||
data view "Vanguard logs + metrics" — filebeat-*,metricbeat-* so Discover and Lens have one source
|
||||
|
||||
WHY AN INDEX AND NOT A WEBHOOK
|
||||
On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
|
||||
leaves the stack (webhook, Slack, email, PagerDuty…) is Gold or above and returns
|
||||
403 Action type .webhook is disabled because your basic license does not support it
|
||||
So Kibana cannot talk to Discord. Instead every rule writes a document into ONE index, which is the alert
|
||||
bus, and bin/vanguard-alerts.sh (a host timer) delivers from there to Discord and later to the bot. That is
|
||||
strictly simpler than the webhook route: no Docker-bridge hop, no dumpd endpoint, and alerts are queryable
|
||||
and replayable after the fact because they are just documents.
|
||||
|
||||
THE HEARTBEAT
|
||||
bin/vanguard-deadman.sh alarms when no heartbeat document has arrived recently. It is the only check that
|
||||
catches the ALERTING pipeline itself dying — if Kibana, its rule engine or the connector stops, no ordinary
|
||||
alert can report it, because the thing that would report it is what broke. The rule queries metricbeat-*,
|
||||
which always has recent documents, so it doubles as a Metricbeat liveness check.
|
||||
|
||||
Run it with an Elasticsearch API key: Basic is not an accepted scheme for this Kibana (the anonymous provider
|
||||
is the only one enabled, so only ApiKey and Bearer are taken).
|
||||
|
||||
pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
|
||||
KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
|
||||
-H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
|
||||
| python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
|
||||
KEY="$KEY" python3 observability/kibana-setup.py
|
||||
|
||||
Order matters: run this AFTER security is on. A rule created while security was off holds no API key and
|
||||
stops working the moment it is enabled, until it is disabled and re-enabled.
|
||||
|
||||
ADDING A REAL ALARM: author it in Kibana as usual, point its action at the "Vanguard alert index" connector,
|
||||
and give the document a message plus a level (warn|error). Do NOT set kind=heartbeat — that value is how the
|
||||
delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
|
||||
for real alarms; only the heartbeat wants to fire every interval.
|
||||
"""
|
||||
import json, os, time, urllib.error, urllib.request
|
||||
|
||||
KIBANA = "http://127.0.0.1:5601"
|
||||
ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
|
||||
HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
|
||||
"Authorization": "ApiKey " + os.environ["KEY"]}
|
||||
|
||||
def wait_for_kibana(seconds=300):
|
||||
"""Kibana has to be *available*, not merely listening.
|
||||
|
||||
configure.py --es-security restarts Kibana as its last step, so anything that calls the Kibana API
|
||||
immediately afterwards races its boot and gets a connection reset. Kibana also answers /api/status while
|
||||
still starting, so the status level is what has to be checked, not the port.
|
||||
"""
|
||||
deadline = time.time() + seconds
|
||||
announced = False
|
||||
while time.time() < deadline:
|
||||
try:
|
||||
with urllib.request.urlopen(KIBANA + "/api/status", timeout=10) as response:
|
||||
level = json.load(response).get("status", {}).get("overall", {}).get("level")
|
||||
if level == "available":
|
||||
print("kibana: available")
|
||||
return
|
||||
state = level or "no level"
|
||||
except Exception as error:
|
||||
state = type(error).__name__
|
||||
if not announced:
|
||||
print(f"kibana: waiting for it to become available ({state})…")
|
||||
announced = True
|
||||
time.sleep(5)
|
||||
raise SystemExit(f"kibana did not become available within {seconds}s — nothing was created")
|
||||
|
||||
def req(method, path, body=None):
|
||||
request = urllib.request.Request(KIBANA + path, method=method,
|
||||
data=None if body is None else json.dumps(body).encode())
|
||||
for key, value in HEADERS.items():
|
||||
request.add_header(key, value)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
raw = response.read()
|
||||
return json.loads(raw) if raw else {}
|
||||
except urllib.error.HTTPError as error:
|
||||
print(f" {method} {path} -> HTTP {error.code}")
|
||||
print(" " + error.read().decode(errors="replace")[:600])
|
||||
raise SystemExit(1)
|
||||
|
||||
wait_for_kibana()
|
||||
|
||||
# 1. the alert bus connector
|
||||
CONNECTOR_NAME = "Vanguard alert index"
|
||||
existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
|
||||
if CONNECTOR_NAME in existing:
|
||||
connector_id = existing[CONNECTOR_NAME]
|
||||
print("connector already present:", connector_id)
|
||||
else:
|
||||
connector_id = req("POST", "/api/actions/connector", {
|
||||
"name": CONNECTOR_NAME,
|
||||
"connector_type_id": ".index",
|
||||
# executionTimeField stamps every document, so a rule body never has to get the timestamp right.
|
||||
"config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
|
||||
"secrets": {}})["id"]
|
||||
print("connector created:", connector_id, "->", ALERT_INDEX)
|
||||
|
||||
# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
|
||||
# harmless and meant the data view was never created on any system where the rule already existed, which
|
||||
# is every re-run.
|
||||
RULE_NAME = "Vanguard alerting heartbeat"
|
||||
existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
|
||||
if r["name"] == RULE_NAME), None)
|
||||
if existing_rule:
|
||||
print("rule already present:", existing_rule["id"],
|
||||
"enabled" if existing_rule.get("enabled") else "DISABLED")
|
||||
else:
|
||||
rule = req("POST", "/api/alerting/rule", {
|
||||
"name": RULE_NAME,
|
||||
"rule_type_id": ".es-query",
|
||||
"consumer": "stackAlerts",
|
||||
"enabled": True,
|
||||
"schedule": {"interval": "1m"},
|
||||
"params": {
|
||||
"searchType": "esQuery",
|
||||
"index": ["metricbeat-*"],
|
||||
"timeField": "@timestamp",
|
||||
"esQuery": json.dumps({"query": {"match_all": {}}}),
|
||||
"size": 1,
|
||||
"aggType": "count",
|
||||
"groupBy": "all",
|
||||
"thresholdComparator": ">",
|
||||
"threshold": [0],
|
||||
"timeWindowSize": 5,
|
||||
"timeWindowUnit": "m",
|
||||
# re-evaluate the whole window every run, so the rule stays active instead of going quiet once
|
||||
# it has already seen those documents
|
||||
"excludeHitsFromPreviousRun": False,
|
||||
},
|
||||
# onActiveAlert writes on EVERY check. Kibana's default only acts when a rule CHANGES status, which
|
||||
# would produce one heartbeat and then silence — the deadman would read that as a dead pipeline.
|
||||
"actions": [{
|
||||
"group": "query matched",
|
||||
"id": connector_id,
|
||||
"params": {"documents": [{
|
||||
"kind": "heartbeat", # the delivery script drops these; never reach Discord
|
||||
"rule": "{{rule.name}}",
|
||||
"level": "info",
|
||||
"message": "kibana rule engine and the index connector are alive",
|
||||
}]},
|
||||
"frequency": {"summary": False, "notify_when": "onActiveAlert", "throttle": None},
|
||||
}],
|
||||
})
|
||||
print("rule created:", rule["id"], "| enabled:", rule.get("enabled"))
|
||||
|
||||
# 3. the data view. Without one, Discover and Lens have nothing to point at and every panel starts with a
|
||||
# detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
|
||||
# metrics, which is the whole reason they share a cluster.
|
||||
VIEW_TITLE = "filebeat-*,metricbeat-*"
|
||||
views = req("GET", "/api/data_views").get("data_view", [])
|
||||
if any(v.get("title") == VIEW_TITLE for v in views):
|
||||
print("data view already present:", VIEW_TITLE)
|
||||
else:
|
||||
view = req("POST", "/api/data_views/data_view", {
|
||||
"data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
|
||||
print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)
|
||||
41
observability/kibana.yml
Normal file
41
observability/kibana.yml
Normal file
@@ -0,0 +1,41 @@
|
||||
# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
|
||||
# into .env, so this file is committed and contains none.
|
||||
#
|
||||
# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
|
||||
# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
|
||||
# whether the current user holds a privilege. With security off that API does not exist, and the
|
||||
# Infrastructure / Metrics app fails on every page view with
|
||||
# no handler found for uri [/_security/user/_has_privileges]
|
||||
# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
|
||||
# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
|
||||
# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
|
||||
# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
|
||||
#
|
||||
# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
|
||||
# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
|
||||
# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
|
||||
# scheme and have Caddy inject a key chosen by Authentik group).
|
||||
|
||||
server.host: "0.0.0.0"
|
||||
server.publicBaseUrl: "https://${DIAG_HOST}"
|
||||
# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
|
||||
# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
|
||||
# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
|
||||
telemetry.optIn: false
|
||||
|
||||
elasticsearch.hosts: ["http://elasticsearch:9200"]
|
||||
elasticsearch.username: "kibana_system" # Kibana's own internal account, NOT what visitors become
|
||||
elasticsearch.password: "${ES_KIBANA_PASSWORD}"
|
||||
|
||||
# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
|
||||
xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
||||
xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
||||
xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
|
||||
|
||||
# The only provider, so there is no login page and no login selector: every request is already signed in.
|
||||
xpack.security.authc.providers:
|
||||
anonymous.anonymous1:
|
||||
order: 0
|
||||
credentials:
|
||||
username: "${VG_DIAG_USER}"
|
||||
password: "${VG_DIAG_PASSWORD}"
|
||||
@@ -27,6 +27,15 @@ metricbeat.modules:
|
||||
metrics_path: /metrics
|
||||
hosts: ["http://127.0.0.1:2019"]
|
||||
|
||||
- module: prometheus # dumpd: OUR metrics — normalise queue, encode time, live streams
|
||||
period: 30s
|
||||
metrics_path: /metrics
|
||||
hosts: ["http://127.0.0.1:8080"] # dumpd's loopback listener; Caddy routes no host to /metrics
|
||||
# These are real counters, so let them arrive typed: Kibana then gets .rate (per-second) as well as the
|
||||
# raw total, which is what makes "encode seconds per hour" a chart rather than a staircase.
|
||||
use_types: true
|
||||
rate_counters: true
|
||||
|
||||
processors:
|
||||
- add_host_metadata: ~
|
||||
# The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a
|
||||
@@ -43,9 +52,18 @@ processors:
|
||||
|
||||
output.elasticsearch:
|
||||
hosts: ["http://127.0.0.1:9200"]
|
||||
# Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
|
||||
# come from the container environment, which compose fills from .env — no secret is committed here.
|
||||
username: "${ES_BEATS_USER}"
|
||||
password: "${ES_BEATS_PASSWORD}"
|
||||
|
||||
setup.kibana:
|
||||
setup.kibana: # no credentials: Kibana's anonymous provider signs these in
|
||||
host: "http://127.0.0.1:5601"
|
||||
setup.dashboards.enabled: true # the system and docker dashboards, so there is something to look at
|
||||
# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
|
||||
# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
|
||||
setup.template.overwrite: true
|
||||
setup.template.settings:
|
||||
index.number_of_replicas: 0
|
||||
|
||||
logging.level: info
|
||||
|
||||
Reference in New Issue
Block a user