techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Browse Source 
Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 19:58:06 -06:00
parent ef357c6d82
commit b38d40ab0c
20 changed files with 1322 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
LOGGING.md
View File

@@ -1,8 +1,10 @@
# Logging framework
Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI
at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so
adopting it is a collector config rather than a code change.
Decided 2026-11-01. The store was chosen, built and replaced on 2026-11-03: a VictoriaLogs stack went in
first and was torn out the same day for producing no dashboards, so the store is now **Elasticsearch with
Kibana**, served at the root of the `diag.BASE` host (Phase 13 in `CommunityDiscordMedia.md`). Logs arrive via
Filebeat, metrics via Metricbeat, into the same cluster so one dashboard can carry both. Everything upstream
of the store was fixed first, so the swap was a collector config rather than a code change.
## Where the lines come from (the only two taps a collector needs)
@@ -11,21 +13,66 @@ adopting it is a collector config rather than a code change.
| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback.
Filebeat reads both: a docker autodiscover template per container (filestream + the container parser) and a
journald input for our units. Metrics are the same story: see the taps table below, all on loopback.
## Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata)
## Metrics taps (Prometheus text format, loopback only — Metricbeat scrapes them)
| endpoint | what | from |
|---|---|---|
| `http://127.0.0.1:9100/metrics` | host: CPU, RAM, disk space + I/O, network, load, md RAID | node_exporter (host network, pid host) |
| `http://127.0.0.1:9101/metrics` | per container: CPU, memory, network, block I/O — labelled with `vanguard.service` / `vanguard.category` | cAdvisor |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet:
metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against
VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on
`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
Host and per-container metrics (CPU, RAM, disk, network, block I/O, restarts) come from Metricbeat's `system`
and `docker` modules, not from an exporter — node_exporter and cAdvisor were removed when the Elastic stack
went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
### Our own metrics, and how to add one
MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
keeping up", because the sweep is a shell script on a timer. So the scripts keep counters and dumpd renders
them.
- `bin/vgmetrics.sh` is sourced by `normalize-sweep.sh` and `vgdeband.sh`. It accumulates `key value` lines in
`/srv/media/.markers/metrics/<component>.counters` and keeps a `<component>.pid` file for as long as the
script is executing. It renders nothing and serves nothing, so the scripts stay simple.
- `bin/dumpd.py` turns those files into Prometheus text on `GET /metrics`, adds gauges it can compute at
scrape time, and asks the MediaMTX API for the live path list. `COUNTER_MAP` there is the single place that
knows the metric names.
| metric | type | what it answers |
|---|---|---|
| `vanguard_media_pending{kind}` | gauge | finished recordings waiting to be normalised. **The capacity signal**: if this trends up across a night, CPU encoding is not keeping pace |
| `vanguard_media_pending_oldest_seconds` | gauge | how long the oldest one has waited |
| `vanguard_media_recording` | gauge | recordings MediaMTX is still writing (age under `MINS_IDLE`) |
| `vanguard_media_encoding_in_flight` | gauge | ffmpeg output files (`*.part`) open right now |
| `vanguard_media_running{component}` | gauge | 1 while the sweep or vgdeband is executing, by live pid, not a stale marker |
| `vanguard_media_jobs_total{component,verdict,outcome,kind}` | counter | remuxes and encodes finished, split by success and by stream vs clip |
| `vanguard_media_job_seconds_total{component,verdict,kind}` | counter | seconds spent in ffmpeg. Divide by the job count for mean encode time |
| `vanguard_media_runs_total{component}` | counter | script starts. Flat means the timer stopped |
| `vanguard_media_deband_total{component}` | counter | timecode bands cropped off |
| `vanguard_media_last_finish_timestamp_seconds{component}` | gauge | when it last finished |
| `vanguard_media_last_run_seconds{component}` | gauge | how long that run took |
| `vanguard_media_failed_files` | gauge | files parked with a `normalize-failed.log` |
| `vanguard_mediamtx_up` | gauge | did the MediaMTX API answer the scrape |
| `vanguard_streams_active` | gauge | streams publishing now, derived `.webrtc` mirrors excluded |
| `vanguard_streams_mirrors` | gauge | those mirrors, counted separately |
| `vanguard_stream_readers` | gauge | viewers attached across all paths |
To add a counter: pick a key of the form `name:label:label`, call `vgmet_add` with it, and add `name` to
`COUNTER_MAP` with the label names in the same order. Nothing else changes, and a key dumpd does not
recognise is ignored rather than breaking the scrape.
Three rules this design exists to keep. Metrics must never break a sweep, so every write is atomic and every
failure in the helper is silent, and a missing helper degrades to no-ops. A scrape must never disturb the
stack, so dumpd never takes the sweep's flock and the filesystem globs are depth-limited rather than walking
the media tree. And the counters live on disk rather than in the process, because the sweep exits between
runs.
Note for anyone editing `normalize-sweep.sh`: its file loop reads from a process substitution, not a pipe. A
pipe would put the loop in a subshell and every counter set inside it would be discarded on exit.
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
@@ -79,10 +126,70 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
"who downloaded what" audit trail for the files host.
## Alerting: Kibana evaluates, we deliver
A basic Elastic licence allows only two connector types, `.index` and `.server-log`. Everything that leaves
the stack — webhook, Slack, email, PagerDuty — is Gold or above and refuses with
`403 Action type .webhook is disabled because your basic license does not support it`. So Kibana cannot talk
to Discord, and the pipeline is split:
Kibana rule --.index connector--> vanguard-alerts --vanguard-alerts.timer--> Discord webhook
- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat
rule. Run it once, *after* Elasticsearch security is on.
- **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
them, and replayable after the fact.
- **`bin/vanguard-alerts.sh`** runs every minute, delivers anything newer than its cursor, and suppresses a
repeat of the same rule within 15 minutes as a safety net. No webhook configured means journal-only and a
non-zero exit, never a silent drop.
- **`bin/vanguard-deadman.sh`** watches both halves: a `kind=heartbeat` document arriving (Kibana still
evaluating and writing) and `alerts.state` being fresh (something still delivering). Either half dying
means no alarm can arrive, and nothing inside the stack could tell you.
### The rules themselves
`observability/kibana-alarms.py` creates them, so they are reviewable and reproducible rather than clicked in.
All of them repeat every **12 hours** while still true (`notify_when: onThrottleInterval`) and go silent when
the condition clears.
| Rule | Level | Fires when |
|---|---|---|
| Disk free below 15% | error | any real filesystem at ≥85% used. Pseudo filesystems are excluded by type, or every tmpfs would alarm forever |
| Memory free below 15% | warn | `system.memory.actual.used.pct` ≥ 0.85 |
| Discord bot not running | error | **no** metrics from the `discord-bot` container for 5 min (the one rule with a `<` comparator) |
| Discord bot unhealthy | error | the container healthcheck reports `unhealthy`, i.e. the bot's own `/health` says Discord is disconnected or the database does not answer |
| Recordings stuck waiting to be normalised | warn | `vanguard_media_pending_oldest_seconds` > 1 h |
| Normalise failures parked | warn | any file has a `normalize-failed.log` |
The two bot rules answer different questions and both are needed. "Not running" catches a stopped or
hard-crash-looping container. "Unhealthy" catches the worse case: the process is up, the port is open, and the
bot is doing nothing because its gateway or database is gone. That second one only works because the bot's
`/health` is exempt from its own identity middleware and returns 503 rather than 200 when degraded — a
healthcheck carries no Authentik headers, and a 200 with `discord_ready: false` would have made a dead bot
look healthy.
Elasticsearch, Kibana, Beat liveness and the disks are **not** here: that is the deadman's job, outside the
stack. Nothing that depends on Kibana can report that Kibana is dead, so duplicating those as rules would go
quiet in exactly the case that matters.
To add an alarm: author the rule in Kibana, point its action at the "Vanguard alert index" connector, and give
the document a `message` and a `level` of `warn` or `error`. Never set `kind: heartbeat` — that value is how
the delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
for real alarms; only the heartbeat should fire every interval.
## Retention
One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
Set by `configure.py --apply`, so nothing here needs clicking and nothing grows without bound:
| What | How long | Set by |
|---|---|---|
| `filebeat-*`, `metricbeat-*` | `LOG_RETENTION_DAYS`, default 30 | a **delete phase injected into the Beats' own ILM policies**. They ship a rollover but no delete, so indices otherwise grow until the disk watermark turns them read-only — which the deadman reports but cannot undo |
| `vanguard-alerts` | `ALERT_RETENTION_DAYS`, default 90 | `vanguard-alerts.sh`, a `_delete_by_query` at most once every 6 h. A heartbeat document arrives every minute, so this index needs pruning more than it needs ILM |
The delete phase is *injected* rather than written whole, so whatever rollover the Beats chose is preserved
across upgrades. Keeping `auth` and `deploy` lines longer than the rest would mean separate indices and a
second ILM policy; not worth it unless an audit asks.
## As built (Phase 13)
@@ -151,18 +258,55 @@ Discord notifier omitted, so nothing is half-configured and nothing fails to loa
- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
earlier entries are still on the host for `journalctl --boot=-1`.
- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it.
file and mount it, the way the gateway and dock pages are served, instead of inlining it. (The diag host no
longer has such a page at all: Kibana occupies its root.)
- **Filebeat 9 removed the `container` input.** It is not deprecated-but-working: the config is refused per
container with "Container input is deprecated. Use Filestream input with its container parser instead",
and Filebeat then runs happily with zero harvesters and creates no index at all. Container logs go through
`type: filestream` with `parsers: [container: ~]`, and every filestream input needs its own stable `id`.
This is exactly the failure the deadman's "up but not shipping" check exists to catch.
- **`add_docker_metadata` should be told where the container id is.** Its path-sniffing default did not
apply to our filestream paths, so the autodiscover template stamps `container.id` and the processor reads
it via `match_fields: ["container.id"]`. Without the id there are no `container.labels.*`, and without
those the `service.name` / `vanguard.category` promotion silently produces nothing.
- **A single node cannot place a replica**, so any beats index defaulting to one replica holds the cluster at
yellow permanently. Both beats set `setup.template.settings.index.number_of_replicas: 0` with
`setup.template.overwrite: true`; indices created before that need a one-time `_settings` call.
- **Kibana's Observability apps that check privileges cannot work with `xpack.security.enabled: false`.** The
Infrastructure / Metrics app calls `api/infra/host/count` on every view and Elasticsearch answers
`no handler found for uri [/_security/user/_has_privileges]`, surfaced as a red "error while fetching
resource" banner. The fix is to turn security ON, not to avoid the app. Doing so normally adds a password
prompt, which this stack forbids, so Kibana's **anonymous provider** signs every visitor in as one native
user instead (`observability/kibana.yml`). Security being on is also what any future read-only tier needs.
- **Moving a setting from a container environment variable into a config file can break it.** `SERVER_BASEPATH: ""`
was fine for years because the Kibana image drops empty env values, but `server.basePath: ""` in
`kibana.yml` is a validation error (`must start with a slash, don't end with one`) and Kibana refuses to
boot. When migrating env settings into a file, an empty value means "omit the key", not "set it to empty".
- **Metricbeat exits when Kibana is unreachable**, because `setup.dashboards.enabled: true` runs at startup.
A Metricbeat restart loop is often a symptom of a Kibana problem, not a Metricbeat one.
- **Alerting rules created while security was off hold no API key** and stop working once it is on, until each
rule is disabled and re-enabled. Create connectors and rules *after* security is on, not before.
- **Kibana's Observability charts group by `event.dataset`.** Beats do not set it, so without a processor
every line lands in one "unknown" bucket and the Logs rate chart is useless. Filebeat now fills it from the
container's `vanguard.service` label, or from `systemd.unit` for journald lines. `copy_fields` never
overwrites an existing target, so listing several sources for one field means first match wins.
- **Metricbeat field shapes differ per module on purpose.** The MediaMTX and Caddy scrapes are untyped, so a
value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
`use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
`.counter` and `.rate`. Look for the right one before concluding a metric is missing.
- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
## Wiring sketches (the options that were weighed)
## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
- **VictoriaLogs**: single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
(`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
behind the Caddy gate. Weakest at log search.
- **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a
single host's logs.
- **ELK** (chosen, and what is deployed): Elasticsearch with a pinned 2 GB heap, Kibana at the root of
`diag.BASE`, Filebeat and Metricbeat shipping into the same cluster so one dashboard can carry logs and
metrics together. The cost is the JVM heap and the index disk on a box that also runs x264; it was taken
because every lighter option needed three or four more pieces to reach a usable dashboard.