diff --git a/LOGGING.md b/LOGGING.md index d5387c1..d5f1f24 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -1,8 +1,10 @@ # Logging framework -Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI -at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so -adopting it is a collector config rather than a code change. +Decided 2026-11-01. The store was chosen, built and replaced on 2026-11-03: a VictoriaLogs stack went in +first and was torn out the same day for producing no dashboards, so the store is now **Elasticsearch with +Kibana**, served at the root of the `diag.BASE` host (Phase 13 in `CommunityDiscordMedia.md`). Logs arrive via +Filebeat, metrics via Metricbeat, into the same cluster so one dashboard can carry both. Everything upstream +of the store was fixed first, so the swap was a collector config rather than a code change. ## Where the lines come from (the only two taps a collector needs) @@ -11,21 +13,66 @@ adopting it is a collector config rather than a code change. | **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u `; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin | | **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs `; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free | -Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback. +Filebeat reads both: a docker autodiscover template per container (filestream + the container parser) and a +journald input for our units. Metrics are the same story: see the taps table below, all on loopback. -## Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata) +## Metrics taps (Prometheus text format, loopback only — Metricbeat scrapes them) | endpoint | what | from | |---|---|---| -| `http://127.0.0.1:9100/metrics` | host: CPU, RAM, disk space + I/O, network, load, md RAID | node_exporter (host network, pid host) | -| `http://127.0.0.1:9101/metrics` | per container: CPU, memory, network, block I/O — labelled with `vanguard.service` / `vanguard.category` | cAdvisor | -| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` | +| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` | | `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) | +| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` | -None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet: -metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against -VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on -`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path. +Host and per-container metrics (CPU, RAM, disk, network, block I/O, restarts) come from Metricbeat's `system` +and `docker` modules, not from an exporter — node_exporter and cAdvisor were removed when the Elastic stack +went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of +them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher). + +### Our own metrics, and how to add one + +MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising +keeping up", because the sweep is a shell script on a timer. So the scripts keep counters and dumpd renders +them. + +- `bin/vgmetrics.sh` is sourced by `normalize-sweep.sh` and `vgdeband.sh`. It accumulates `key value` lines in + `/srv/media/.markers/metrics/.counters` and keeps a `.pid` file for as long as the + script is executing. It renders nothing and serves nothing, so the scripts stay simple. +- `bin/dumpd.py` turns those files into Prometheus text on `GET /metrics`, adds gauges it can compute at + scrape time, and asks the MediaMTX API for the live path list. `COUNTER_MAP` there is the single place that + knows the metric names. + +| metric | type | what it answers | +|---|---|---| +| `vanguard_media_pending{kind}` | gauge | finished recordings waiting to be normalised. **The capacity signal**: if this trends up across a night, CPU encoding is not keeping pace | +| `vanguard_media_pending_oldest_seconds` | gauge | how long the oldest one has waited | +| `vanguard_media_recording` | gauge | recordings MediaMTX is still writing (age under `MINS_IDLE`) | +| `vanguard_media_encoding_in_flight` | gauge | ffmpeg output files (`*.part`) open right now | +| `vanguard_media_running{component}` | gauge | 1 while the sweep or vgdeband is executing, by live pid, not a stale marker | +| `vanguard_media_jobs_total{component,verdict,outcome,kind}` | counter | remuxes and encodes finished, split by success and by stream vs clip | +| `vanguard_media_job_seconds_total{component,verdict,kind}` | counter | seconds spent in ffmpeg. Divide by the job count for mean encode time | +| `vanguard_media_runs_total{component}` | counter | script starts. Flat means the timer stopped | +| `vanguard_media_deband_total{component}` | counter | timecode bands cropped off | +| `vanguard_media_last_finish_timestamp_seconds{component}` | gauge | when it last finished | +| `vanguard_media_last_run_seconds{component}` | gauge | how long that run took | +| `vanguard_media_failed_files` | gauge | files parked with a `normalize-failed.log` | +| `vanguard_mediamtx_up` | gauge | did the MediaMTX API answer the scrape | +| `vanguard_streams_active` | gauge | streams publishing now, derived `.webrtc` mirrors excluded | +| `vanguard_streams_mirrors` | gauge | those mirrors, counted separately | +| `vanguard_stream_readers` | gauge | viewers attached across all paths | + +To add a counter: pick a key of the form `name:label:label`, call `vgmet_add` with it, and add `name` to +`COUNTER_MAP` with the label names in the same order. Nothing else changes, and a key dumpd does not +recognise is ignored rather than breaking the scrape. + +Three rules this design exists to keep. Metrics must never break a sweep, so every write is atomic and every +failure in the helper is silent, and a missing helper degrades to no-ops. A scrape must never disturb the +stack, so dumpd never takes the sweep's flock and the filesystem globs are depth-limited rather than walking +the media tree. And the counters live on disk rather than in the process, because the sweep exits between +runs. + +Note for anyone editing `normalize-sweep.sh`: its file loop reads from a process substitution, not a pipe. A +pipe would put the loop in a subshell and every counter set inside it would be discarded on exit. ## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`) @@ -79,10 +126,70 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie `X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the "who downloaded what" audit trail for the files host. +## Alerting: Kibana evaluates, we deliver + +A basic Elastic licence allows only two connector types, `.index` and `.server-log`. Everything that leaves +the stack — webhook, Slack, email, PagerDuty — is Gold or above and refuses with +`403 Action type .webhook is disabled because your basic license does not support it`. So Kibana cannot talk +to Discord, and the pipeline is split: + + Kibana rule --.index connector--> vanguard-alerts --vanguard-alerts.timer--> Discord webhook + +- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat + rule. Run it once, *after* Elasticsearch security is on. +- **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind` + as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused + them, and replayable after the fact. +- **`bin/vanguard-alerts.sh`** runs every minute, delivers anything newer than its cursor, and suppresses a + repeat of the same rule within 15 minutes as a safety net. No webhook configured means journal-only and a + non-zero exit, never a silent drop. +- **`bin/vanguard-deadman.sh`** watches both halves: a `kind=heartbeat` document arriving (Kibana still + evaluating and writing) and `alerts.state` being fresh (something still delivering). Either half dying + means no alarm can arrive, and nothing inside the stack could tell you. + +### The rules themselves + +`observability/kibana-alarms.py` creates them, so they are reviewable and reproducible rather than clicked in. +All of them repeat every **12 hours** while still true (`notify_when: onThrottleInterval`) and go silent when +the condition clears. + +| Rule | Level | Fires when | +|---|---|---| +| Disk free below 15% | error | any real filesystem at ≥85% used. Pseudo filesystems are excluded by type, or every tmpfs would alarm forever | +| Memory free below 15% | warn | `system.memory.actual.used.pct` ≥ 0.85 | +| Discord bot not running | error | **no** metrics from the `discord-bot` container for 5 min (the one rule with a `<` comparator) | +| Discord bot unhealthy | error | the container healthcheck reports `unhealthy`, i.e. the bot's own `/health` says Discord is disconnected or the database does not answer | +| Recordings stuck waiting to be normalised | warn | `vanguard_media_pending_oldest_seconds` > 1 h | +| Normalise failures parked | warn | any file has a `normalize-failed.log` | + +The two bot rules answer different questions and both are needed. "Not running" catches a stopped or +hard-crash-looping container. "Unhealthy" catches the worse case: the process is up, the port is open, and the +bot is doing nothing because its gateway or database is gone. That second one only works because the bot's +`/health` is exempt from its own identity middleware and returns 503 rather than 200 when degraded — a +healthcheck carries no Authentik headers, and a 200 with `discord_ready: false` would have made a dead bot +look healthy. + +Elasticsearch, Kibana, Beat liveness and the disks are **not** here: that is the deadman's job, outside the +stack. Nothing that depends on Kibana can report that Kibana is dead, so duplicating those as rules would go +quiet in exactly the case that matters. + +To add an alarm: author the rule in Kibana, point its action at the "Vanguard alert index" connector, and give +the document a `message` and a `level` of `warn` or `error`. Never set `kind: heartbeat` — that value is how +the delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle +for real alarms; only the heartbeat should fire every interval. + ## Retention -One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping -`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does. +Set by `configure.py --apply`, so nothing here needs clicking and nothing grows without bound: + +| What | How long | Set by | +|---|---|---| +| `filebeat-*`, `metricbeat-*` | `LOG_RETENTION_DAYS`, default 30 | a **delete phase injected into the Beats' own ILM policies**. They ship a rollover but no delete, so indices otherwise grow until the disk watermark turns them read-only — which the deadman reports but cannot undo | +| `vanguard-alerts` | `ALERT_RETENTION_DAYS`, default 90 | `vanguard-alerts.sh`, a `_delete_by_query` at most once every 6 h. A heartbeat document arrives every minute, so this index needs pruning more than it needs ILM | + +The delete phase is *injected* rather than written whole, so whatever rollover the Beats chose is preserved +across upgrades. Keeping `auth` and `deploy` lines longer than the rest would mean separate indices and a +second ILM policy; not worth it unless an audit asks. ## As built (Phase 13) @@ -151,18 +258,55 @@ Discord notifier omitted, so nothing is half-configured and nothing fails to loa - **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped; earlier entries are still on the host for `journalctl --boot=-1`. - **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a - file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it. + file and mount it, the way the gateway and dock pages are served, instead of inlining it. (The diag host no + longer has such a page at all: Kibana occupies its root.) +- **Filebeat 9 removed the `container` input.** It is not deprecated-but-working: the config is refused per + container with "Container input is deprecated. Use Filestream input with its container parser instead", + and Filebeat then runs happily with zero harvesters and creates no index at all. Container logs go through + `type: filestream` with `parsers: [container: ~]`, and every filestream input needs its own stable `id`. + This is exactly the failure the deadman's "up but not shipping" check exists to catch. +- **`add_docker_metadata` should be told where the container id is.** Its path-sniffing default did not + apply to our filestream paths, so the autodiscover template stamps `container.id` and the processor reads + it via `match_fields: ["container.id"]`. Without the id there are no `container.labels.*`, and without + those the `service.name` / `vanguard.category` promotion silently produces nothing. +- **A single node cannot place a replica**, so any beats index defaulting to one replica holds the cluster at + yellow permanently. Both beats set `setup.template.settings.index.number_of_replicas: 0` with + `setup.template.overwrite: true`; indices created before that need a one-time `_settings` call. +- **Kibana's Observability apps that check privileges cannot work with `xpack.security.enabled: false`.** The + Infrastructure / Metrics app calls `api/infra/host/count` on every view and Elasticsearch answers + `no handler found for uri [/_security/user/_has_privileges]`, surfaced as a red "error while fetching + resource" banner. The fix is to turn security ON, not to avoid the app. Doing so normally adds a password + prompt, which this stack forbids, so Kibana's **anonymous provider** signs every visitor in as one native + user instead (`observability/kibana.yml`). Security being on is also what any future read-only tier needs. +- **Moving a setting from a container environment variable into a config file can break it.** `SERVER_BASEPATH: ""` + was fine for years because the Kibana image drops empty env values, but `server.basePath: ""` in + `kibana.yml` is a validation error (`must start with a slash, don't end with one`) and Kibana refuses to + boot. When migrating env settings into a file, an empty value means "omit the key", not "set it to empty". +- **Metricbeat exits when Kibana is unreachable**, because `setup.dashboards.enabled: true` runs at startup. + A Metricbeat restart loop is often a symptom of a Kibana problem, not a Metricbeat one. +- **Alerting rules created while security was off hold no API key** and stop working once it is on, until each + rule is disabled and re-enabled. Create connectors and rules *after* security is on, not before. +- **Kibana's Observability charts group by `event.dataset`.** Beats do not set it, so without a processor + every line lands in one "unknown" bucket and the Logs rate chart is useless. Filebeat now fills it from the + container's `vanguard.service` label, or from `systemd.unit` for journald lines. `copy_fields` never + overwrites an existing target, so listing several sources for one field means first match wins. +- **Metricbeat field shapes differ per module on purpose.** The MediaMTX and Caddy scrapes are untyped, so a + value is at `prometheus.metrics.` with labels at `prometheus.labels.*`. The dumpd scrape sets + `use_types` and `rate_counters`, so a gauge is at `prometheus..value` and a counter has both + `.counter` and `.rate`. Look for the right one before concluding a metric is missing. - **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D. -## Wiring sketches (the options that were weighed) +## Wiring sketches (the options that were weighed, kept so the decision is not re-argued) - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea). Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana. -- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector +- **VictoriaLogs**: single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE` (`media-admins`). VictoriaMetrics beside it would consume the four metric taps. - **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer; behind the Caddy gate. Weakest at log search. -- **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a - single host's logs. +- **ELK** (chosen, and what is deployed): Elasticsearch with a pinned 2 GB heap, Kibana at the root of + `diag.BASE`, Filebeat and Metricbeat shipping into the same cluster so one dashboard can carry logs and + metrics together. The cost is the JVM heap and the index disk on a box that also runs x264; it was taken + because every lighter option needed three or four more pieces to reach a usable dashboard. diff --git a/STATUS.md b/STATUS.md index cfbac58..3eaee0c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # Vanguard stack — what runs, what people do, what's unfinished -Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01. +Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03. Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box. Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`). @@ -59,8 +59,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | normalize sweep | editing standard, one copy per clip | live | | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live | | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) | -| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) | -| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** | +| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) | +| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** | +| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything | +| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy | +| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** | | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending | | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live | | Jellyfin | watch page, SSO via Authentik, direct play only | live | @@ -91,6 +94,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea 9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served from the bot itself now; an Authentik username is matched to one Discord member by name. +9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics. +9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid. 10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs. 11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box. @@ -111,3 +116,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested). - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply. - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only. +- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields. +- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`. +- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading. +- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason. +- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work. diff --git a/bin/dumpd.py b/bin/dumpd.py index 1c6b406..179bc1a 100755 --- a/bin/dumpd.py +++ b/bin/dumpd.py @@ -12,12 +12,17 @@ POST /me/clips -> {"device_id": "..."}: create clips/ (+ .stignore), add POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from /etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1; otherwise logs the push and answers 202 "auto-deploy disabled". -POST /heartbeat -> Alertmanager's Watchdog route: touches a file the deadman timer checks, so a dead - alerting pipeline becomes visible (bin/vanguard-deadman.sh). +POST /heartbeat -> LEGACY, nothing calls this now. It was the Watchdog route when Alertmanager, and then a + Kibana webhook connector, posted here. A basic Elastic licence has no webhook connector, so + Kibana's rules write into the vanguard-alerts index instead and the deadman reads the + heartbeat from there. Kept because it costs nothing and is a ready-made target if an + external monitor is ever added. POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle, HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot. May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body. +GET /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live + gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here. POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret. @@ -420,6 +425,140 @@ def verify_token(token): return None return user.lower() +# ---- GET /metrics ----------------------------------------------------------------------------------- +# Prometheus text for Metricbeat's prometheus module. Two sources are merged: +# 1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/.counters) +# 2. gauges computed here at scrape time: unpublished recordings, encodes in flight, failures, MediaMTX +# Only the loopback listener serves it, and Caddy routes no host to /metrics (every dumpd route over there is +# an explicit path matcher), so this is reachable from the host and from nowhere else. Nothing here is +# authenticated and nothing here is allowed to fail: a scrape must never be able to disturb the stack. +METRICS_DIR = Path(os.environ.get("VG_METRICS_DIR", str(MARK / "metrics"))) +MTX_API = os.environ.get("MTX_API_URL", "http://127.0.0.1:9997") +IDLE_S = int(os.environ.get("MINS_IDLE", "3")) * 60 # the sweep's own "finished being written" rule + +# counters key head -> (metric, type, help, the labels the remaining key fields fill, in order) +COUNTER_MAP = { + "jobs": ("vanguard_media_jobs_total", "counter", "normalise/deband jobs that finished", ("verdict", "outcome", "kind")), + "seconds": ("vanguard_media_job_seconds_total", "counter", "wall-clock seconds spent in ffmpeg", ("verdict", "kind")), + "runs": ("vanguard_media_runs_total", "counter", "times the script has started work", ()), + "deband": ("vanguard_media_deband_total", "counter", "timecode bands cropped off", ()), + "last_finish": ("vanguard_media_last_finish_timestamp_seconds", "gauge", "when the script last finished", ()), + "last_duration": ("vanguard_media_last_run_seconds", "gauge", "how long the last finished run took", ()), +} + +def _num(value): + """Prometheus wants a plain number: %g would turn a unix timestamp into 1.76e+09.""" + return "%d" % value if float(value).is_integer() else "%.3f" % value + +def _glob(*patterns): + found = [] + for pattern in patterns: + try: + found.extend(ROOT.glob(pattern)) + except OSError: + pass + return found + +def mtx_paths(): + """MediaMTX's own view of what is live. None if it did not answer.""" + try: + with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=1.5) as response: + return json.load(response).get("items") or [] + except Exception: + return None + +def metrics_text(): + order, grouped = [], {} + def add(metric, typ, helptext, labels, value): + if metric not in grouped: + grouped[metric] = (typ, helptext, []) + order.append(metric) + grouped[metric][2].append((labels, value)) + + # 1. whatever the scripts have counted + for counters in sorted(METRICS_DIR.glob("*.counters")): + component = counters.name[: -len(".counters")] + try: + lines = counters.read_text().splitlines() + except OSError: + continue + for line in lines: + field = line.split() + if len(field) != 2: + continue + try: + value = float(field[1]) + except ValueError: + continue + head, *rest = field[0].split(":") + if head not in COUNTER_MAP: + continue + metric, typ, helptext, names = COUNTER_MAP[head] + labels = {"component": component} + labels.update(dict(zip(names, rest))) + add(metric, typ, helptext, labels, value) + + # 2. is each script executing right now? (its pid file exists only while it runs; a crash leaves a stale + # one, so the pid is checked against /proc rather than trusted) + for pid_file in sorted(METRICS_DIR.glob("*.pid")): + try: + pid = int(pid_file.read_text().strip() or 0) + except (OSError, ValueError): + pid = 0 + add("vanguard_media_running", "gauge", "1 while the script is executing", + {"component": pid_file.name[: -len(".pid")]}, 1 if pid and Path(f"/proc/{pid}").is_dir() else 0) + + # 3. the queue. Depth-limited globs on purpose: never a walk of the whole media tree. + now = _time.time() + recording, pending, oldest = 0, 0, 0.0 + for f in _glob("streams/*/*/.live/*"): + if f.name.startswith(".") or not f.is_file(): + continue + try: + age = now - f.stat().st_mtime + except OSError: + continue + # the sweep ignores anything touched in the last MINS_IDLE, so younger files are still being written + if age <= IDLE_S: + recording += 1 + else: + pending += 1 + oldest = max(oldest, age) + add("vanguard_media_recording", "gauge", "recordings MediaMTX is still writing into .live/", {}, recording) + add("vanguard_media_pending", "gauge", "finished recordings waiting to be normalised", {"kind": "stream"}, pending) + add("vanguard_media_pending_oldest_seconds", "gauge", "age of the oldest waiting recording", {}, round(oldest)) + add("vanguard_media_encoding_in_flight", "gauge", "ffmpeg output files open right now", {}, + len(_glob("streams/*/*.part", "streams/*/*/*.part", "streams/*/*/.live/*.part", + "clips/*/*.part", "clips/*/*/*.part", "clips/*/*/*/*.part"))) + add("vanguard_media_failed_files", "gauge", "files parked with a normalize-failed log", {}, + len(_glob("streams/*/*.normalize-failed.log", "streams/*/*/*.normalize-failed.log", + "clips/*/*.normalize-failed.log", "clips/*/*/*.normalize-failed.log", + "clips/*/*/*/*.normalize-failed.log"))) + + # 4. MediaMTX. The Prometheus scrape of :9998 already has per-path detail; this is the count the portal + # and the dock actually mean by "live", with the derived .webrtc audio mirrors separated out. + paths = mtx_paths() + add("vanguard_mediamtx_up", "gauge", "did the MediaMTX API answer this scrape", {}, 0 if paths is None else 1) + if paths is not None: + ready = [p for p in paths if p.get("ready")] + add("vanguard_streams_active", "gauge", "streams publishing right now, mirrors excluded", {}, + sum(1 for p in ready if not str(p.get("name", "")).endswith(".webrtc"))) + add("vanguard_streams_mirrors", "gauge", "derived .webrtc audio mirrors running", {}, + sum(1 for p in ready if str(p.get("name", "")).endswith(".webrtc"))) + add("vanguard_stream_readers", "gauge", "viewers attached across all paths", {}, + sum(len(p.get("readers") or []) for p in ready)) + + out = [] + for metric in order: + typ, helptext, series = grouped[metric] + if helptext: + out.append(f"# HELP {metric} {helptext}") + out.append(f"# TYPE {metric} {typ}") + for labels, value in series: + tags = ",".join(f'{k}="{v}"' for k, v in sorted(labels.items())) + out.append(f"{metric}{{{tags}}} {_num(value)}" if tags else f"{metric} {_num(value)}") + return "\n".join(out) + "\n" + class H(BaseHTTPRequestHandler): server_version = "dumpd/1" def _json(self, code, obj): @@ -429,6 +568,13 @@ class H(BaseHTTPRequestHandler): self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) + def _text(self, code, body): + raw = body.encode() + self.send_response(code) + self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) def _deny(self): self._json(401, {"error": "not authenticated (no identity header from the proxy)"}) def _parse(self): @@ -457,6 +603,12 @@ class H(BaseHTTPRequestHandler): def do_GET(self): if self.server.server_port == AUTH_LISTEN[1]: return self._json(404, {"error": "not found"}) # the auth port serves POST /mtx-auth only + if self.path == "/metrics": + try: + return self._text(200, metrics_text()) + except Exception as error: # a scrape must never take dumpd down + log("metrics_failed", level="error", error=str(error)) + return self._text(500, f"# metrics unavailable: {error}\n") if self.path.startswith("/time"): # server clock for the timecode strip (the /clock/ page and Go Live measure their offset against this) return self._json(200, {"now_ms": int(_time.time() * 1000)}) diff --git a/bin/normalize-sweep.sh b/bin/normalize-sweep.sh index 09aff87..ba2f15e 100755 --- a/bin/normalize-sweep.sh +++ b/bin/normalize-sweep.sh @@ -36,6 +36,15 @@ MAX_W=2560; MAX_H=1440; MAX_FPS=60 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}" flock -n 9 || exit 0 +# Metrics (LOGGING.md): counters for dumpd's /metrics. Optional on purpose — if the helper is not installed +# yet, these become no-ops and normalising carries on exactly as before. +if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then + VG_MET_COMPONENT=sweep + . "$(dirname "$0")/vgmetrics.sh" && vgmet_init +else + vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; } +fi + # logfmt (LOGGING.md): vglog event= key=value ... values with spaces must arrive pre-quoted vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=sweep %s\n' "$(date -Is)" "$lvl" "$*"; } log() { vglog info "msg=\"$*\""; } @@ -84,9 +93,6 @@ probe() { echo "ok" } -find "$ROOT/streams" "$ROOT/clips" -type f \ - \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \ - ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null | while IFS= read -r -d '' src; do base="${src%.*}"; ext="${src##*.}" live=0; stream="" @@ -121,6 +127,7 @@ while IFS= read -r -d '' src; do [[ -e "$out" ]] && continue fi tmp="$out.part" + kind=clip; (( live )) && kind=stream # metric label: stream encodes are the expensive ones vglog info "event=verdict verdict=$verdict file=\"$src\" reason=\"$reason\" stream=\"$stream\"" tcargs=(); tcjson=""; band=0; deband=0 if (( live )); then @@ -157,9 +164,11 @@ PYEOF fi fi fi + job_t0=$(vgmet_time) + err=$(mktemp /tmp/vgsweep-err.XXXXXX) # ffmpeg's own words; rc= alone is not diagnosable if [[ $verdict == remux ]]; then nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \ - -map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" + -map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err" else # scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2" @@ -175,11 +184,14 @@ PYEOF -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \ -vf "$vf" -fps_mode "$fpsmode" "${fpsargs[@]}" \ "${aargs[@]}" \ - "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" + "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err" fi rc=$? + job_secs=$(vgmet_since "$job_t0") d1=$(dur "$src"); d2=$(dur "$tmp") if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then + vgmet_add "jobs:$verdict:done:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs" + (( deband )) && vgmet_add deband 1 if (( live )); then mv -f "$tmp" "$out"; chown 1000:1000 "$out" 2>/dev/null if [[ -n $tcjson ]]; then # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples @@ -192,6 +204,7 @@ PYEOF rm -f "$src" # band removal: the cropped file IS the recording; raw-with-band is not kept fi rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true # drop .live and the empty title folder + rm -f "$err" vglog info "event=published file=\"$out\" stream=\"$stream\" verdict=$verdict"; continue elif (( synced )); then rel_src="${src#"$stroot"/}"; rel_out="${out#"$stroot"/}" @@ -211,7 +224,13 @@ PYEOF vglog info "event=done file=\"$out\" verdict=$verdict" else rm -f "$tmp" - { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"; } > "$base.normalize-failed.log" - vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc log=\"$base.normalize-failed.log\"" >&2 + vgmet_add "jobs:$verdict:failed:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs" + { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)" + echo "--- ffmpeg stderr ---"; tail -40 "$err" 2>/dev/null; } > "$base.normalize-failed.log" + why=$(tail -1 "$err" 2>/dev/null | tr -d '"' | cut -c1-200) + vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc reason=\"${why:-no stderr}\" log=\"$base.normalize-failed.log\"" >&2 fi -done + rm -f "$err" +done < <(find "$ROOT/streams" "$ROOT/clips" -type f \ + \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \ + ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null) diff --git a/bin/vanguard-alerts.sh b/bin/vanguard-alerts.sh new file mode 100755 index 0000000..385ca66 --- /dev/null +++ b/bin/vanguard-alerts.sh @@ -0,0 +1,152 @@ +#!/bin/bash +# Deliver Kibana's alerts to Discord. Runs from a systemd timer every minute; curl and python only. +# +# WHY THIS EXISTS +# On a basic license Kibana's only connector types are .index and .server-log — every connector that leaves +# the stack is Gold or above. So Kibana rules write their alerts into ONE Elasticsearch index (the alert bus, +# created by observability/kibana-setup.py) and this script is the delivery side. Alerts are therefore +# ordinary documents: queryable, replayable, and visible in Discover alongside the logs that caused them. +# +# Heartbeat documents (kind=heartbeat) are deliberately NOT delivered. They exist for bin/vanguard-deadman.sh +# to notice their absence, which is the only way to catch the alerting pipeline itself dying. +# +# Repeat suppression is a safety net, not the primary throttle: a rule should use "On status change" or its +# own throttle in Kibana. This stops a misconfigured rule turning into a message every minute, and it is per +# rule name, so one noisy rule never silences another. +# +# Exit status: 0 when there was nothing to send or everything was sent; 1 when alerts existed and could not be +# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure. +set -uo pipefail + +ES="${ES_URL:-http://127.0.0.1:9200}" +ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}" +INDEX="${ALERT_INDEX:-vanguard-alerts}" +STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}" +WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}" +SUPPRESS_S="${ALERT_SUPPRESS_S:-900}" +BATCH="${ALERT_BATCH:-50}" + +vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=alerts %s\n' "$(date -Is)" "$lvl" "$*"; } +es_curl() { if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi; } + +state_get() { [[ -r $STATE ]] || return 0; grep -m1 "^$1=" "$STATE" 2>/dev/null | cut -d= -f2-; } +state_set() { + local key="$1" value="$2" tmp + install -d "$(dirname "$STATE")" 2>/dev/null || return 0 + tmp=$(mktemp "${STATE}.XXXXXX" 2>/dev/null) || return 0 + { [[ -r $STATE ]] && grep -v "^${key}=" "$STATE"; printf '%s=%s\n' "$key" "$value"; } > "$tmp" 2>/dev/null + mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp" +} + +webhook="" +[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE") +# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like +# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit. +case "${webhook:-}" in + "") ;; + https://*) ;; + *) vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;; +esac + +now=$(date +%s) +cursor=$(state_get cursor); cursor="${cursor:-0}" + +# Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely. +query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor") +response=$(es_curl -m 15 -H 'Content-Type: application/json' "$ES/$INDEX/_search" -d "$query") + +if [[ -z $response ]]; then + vglog error 'event=unreachable msg="Elasticsearch did not answer; cannot read the alert bus"' >&2 + exit 1 +fi +# A missing index just means no rule has fired yet. Not a fault, and not worth a line every minute. +if grep -q 'index_not_found_exception' <<<"$response"; then + exit 0 +fi + +# Retention. A heartbeat document arrives every minute, so without this the alert bus grows without bound. +# At most once every 6 h: a delete_by_query is cheap, but not worth doing sixty times an hour. Deliberately +# before the "nothing new" exit below, because most runs have nothing to deliver and would skip it. +retention_days="${ALERT_RETENTION_DAYS:-90}" +last_purge=$(state_get purged); last_purge="${last_purge:-0}" +if (( now - last_purge > 21600 )); then + purge=$(es_curl -m 120 -H 'Content-Type: application/json' \ + -XPOST "$ES/$INDEX/_delete_by_query?conflicts=proceed" \ + -d "{\"query\":{\"range\":{\"@timestamp\":{\"lt\":\"now-${retention_days}d\"}}}}") + gone=$(printf '%s' "${purge:-}" | python3 -c 'import json,sys +try: print(json.load(sys.stdin).get("deleted", 0)) +except Exception: print("?")') + state_set purged "$now" + vglog info "event=purged older_than=${retention_days}d deleted=${gone:-?}" +fi + +rows=$(printf '%s' "$response" | python3 -c ' +import json, sys +from datetime import datetime +try: + data = json.load(sys.stdin) +except Exception: + sys.exit(0) +for hit in data.get("hits", {}).get("hits", []): + source = hit.get("_source", {}) + stamp = str(source.get("@timestamp", "")) + try: + when = datetime.fromisoformat(stamp.replace("Z", "+00:00")) + millis = int(when.timestamp() * 1000) + except Exception: + continue + def clean(value, limit): + return " ".join(str(value).split())[:limit] or "-" + print("\t".join((str(millis), + clean(source.get("rule", "unnamed rule"), 120), + clean(source.get("level", "warn"), 12), + clean(source.get("message", json.dumps(source)[:400]), 1200)))) +') + +[[ -z $rows ]] && exit 0 + +# One per line: "a=0 b=0 now" would make bash run `now` as a COMMAND with a and b as temporary +# assignments, leaving every counter unset for the rest of the script. +sent=0 +suppressed=0 +failed=0 +while IFS=$'\t' read -r millis rule level message; do + [[ -z ${millis:-} ]] && continue + last=$(state_get "sent:$rule"); last="${last:-0}" + if (( now - last < SUPPRESS_S )); then + suppressed=$((suppressed + 1)) + vglog info "event=suppressed rule=\"$rule\" within=${SUPPRESS_S}s" + state_set cursor "$millis" + continue + fi + case "$level" in + error|critical) mark="[ERROR]" ;; + warn|warning) mark="[WARN]" ;; + *) mark="[$level]" ;; + esac + text="$mark $rule — $message" + if [[ -z $webhook ]]; then + vglog warn "event=undeliverable rule=\"$rule\" msg=\"no Discord webhook configured; alert only in the journal\"" >&2 + vglog warn "event=alert rule=\"$rule\" level=$level msg=\"$message\"" >&2 + failed=$((failed + 1)) + state_set cursor "$millis" + continue + fi + payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))') + # curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000. + code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \ + -d "$payload" "$webhook" || true) + code="${code:-000}" + if [[ $code == 2* ]]; then + sent=$((sent + 1)); state_set "sent:$rule" "$now" + vglog info "event=delivered rule=\"$rule\" level=$level" + else + failed=$((failed + 1)) + vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2 + fi + state_set cursor "$millis" +done <<<"$rows" + +vglog info "event=run sent=$sent suppressed=$suppressed failed=$failed" +(( failed > 0 )) && exit 1 +exit 0 diff --git a/bin/vanguard-deadman.sh b/bin/vanguard-deadman.sh index f40a368..8691200 100755 --- a/bin/vanguard-deadman.sh +++ b/bin/vanguard-deadman.sh @@ -25,9 +25,19 @@ set -uo pipefail ES="${ES_URL:-http://127.0.0.1:9200}" +# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes +# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file +# is missing, the calls below get 401 and the deadman reports Elasticsearch as unreachable — which is correct, +# because from its point of view it is. Kibana needs nothing: its anonymous provider signs callers in. +ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}" +es_curl() { + if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi +} KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}" WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}" -HEARTBEAT="${ALERT_HEARTBEAT_FILE:-/srv/media/.markers/alert-heartbeat}" +ALERT_INDEX="${ALERT_INDEX:-vanguard-alerts}" # the alert bus Kibana's rules write into +ALERT_STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}" # touched by every vanguard-alerts.sh run +ALERT_STATE_MAX_S="${ALERT_STATE_MAX_S:-900}" HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}" STALE_MAX_S="${BEAT_STALE_MAX_S:-900}" STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}" @@ -44,7 +54,7 @@ for k in sys.argv[1].split("."): print(d)' "$1" 2>/dev/null; } # 1. Elasticsearch -health=$(curl -s -m 8 "$ES/_cluster/health" || true) +health=$(es_curl -m 8 "$ES/_cluster/health" || true) if [[ -z $health ]]; then note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored" else @@ -56,7 +66,7 @@ else esac # 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it. - blocked=$(curl -s -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \ + blocked=$(es_curl -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \ | python3 -c 'import json,sys try: d=json.load(sys.stdin) except Exception: sys.exit(0) @@ -66,7 +76,7 @@ print(len(bad))' 2>/dev/null || echo 0) # 4 and 5. are the Beats actually writing? for beat in filebeat metricbeat; do - count=$(curl -s -m 8 -H 'Content-Type: application/json' \ + count=$(es_curl -m 8 -H 'Content-Type: application/json' \ "$ES/${beat}-*/_count" \ -d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count) if [[ -z ${count:-} ]]; then @@ -81,12 +91,31 @@ fi level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level) [[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down" -# 6. the heartbeat a Kibana rule posts here. Missing right after a restart is not yet a fault; age is. -if [[ -f $HEARTBEAT ]]; then - age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") )) - (( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (the Kibana rule → webhook → dumpd path is broken)" -else - note "no alert heartbeat has ever arrived (is the Kibana heartbeat rule created? README 5h)" +# 6. the heartbeat. A Kibana rule writes a kind=heartbeat document into the alert index every minute (see +# observability/kibana-setup.py). Its absence is the ONLY thing that catches the alerting pipeline itself +# dying: if Kibana, its rule engine or the index connector stops, no ordinary alert can report it, because +# the thing that would report it is what broke. Checked in Elasticsearch, not as a file: a basic licence has +# no webhook connector, so nothing posts to dumpd any more. +if [[ -n $health ]]; then + beats=$(es_curl -m 8 -H 'Content-Type: application/json' "$ES/$ALERT_INDEX/_count" \ + -d "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"kind\":\"heartbeat\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-${HEARTBEAT_MAX_S}s\"}}}]}}}") + if grep -q 'index_not_found_exception' <<<"${beats:-}"; then + note "the alert index $ALERT_INDEX does not exist — the Kibana connector and heartbeat rule have not been created (observability/kibana-setup.py)" + else + hb=$(printf '%s' "${beats:-}" | jq_get count) + if [[ -z ${hb:-} ]]; then + note "cannot read the alert heartbeat (query against $ALERT_INDEX failed)" + elif (( hb == 0 )); then + note "no alert heartbeat in the last $((HEARTBEAT_MAX_S/60)) minutes — Kibana's rule engine or the index connector has stopped, so NO alert can reach Discord" + fi + fi +fi + +# 7. is the delivery side running? The heartbeat proves Kibana still writes alerts; this proves something +# still takes them to Discord. Both halves have to be alive for an alarm to arrive. +if [[ -f $ALERT_STATE ]]; then + age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") )) + (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered" fi # the disks, independently of any agent diff --git a/bin/vgdeband.sh b/bin/vgdeband.sh index 827a8c1..2f2726d 100755 --- a/bin/vgdeband.sh +++ b/bin/vgdeband.sh @@ -11,6 +11,13 @@ ROOT="${MEDIA_ROOT:-/srv/media}" vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgdeband %s\n' "$(date -Is)" "$lvl" "$*"; } log() { vglog info "msg=\"$*\""; } dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; } +# Metrics (LOGGING.md): the same counters the sweep keeps, under component=vgdeband. No-ops if not installed. +if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then + VG_MET_COMPONENT=vgdeband + . "$(dirname "$0")/vgmetrics.sh" && vgmet_init +else + vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; } +fi if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi for f in "${files[@]}"; do side="${f%.*}.json" @@ -26,19 +33,22 @@ print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode tmp="$f.part" tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime") vglog info "event=deband file=\"$f\" width=$w band_rows=$band timecode=\"$tc\"" + job_t0=$(vgmet_time) nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \ -map 0:v:0 -map '0:a?' \ -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "${ENCODE_THREADS:-2}" \ -vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \ "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" - rc=$?; d1=$(dur "$f"); d2=$(dur "$tmp") + rc=$?; job_secs=$(vgmet_since "$job_t0"); d1=$(dur "$f"); d2=$(dur "$tmp") + vgmet_add "seconds:encode:stream" "$job_secs" if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then + vgmet_add "jobs:encode:done:stream"; vgmet_add deband 1 mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null python3 -c ' import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null vglog info "event=done file=\"$f\"" else - rm -f "$tmp"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2 + rm -f "$tmp"; vgmet_add "jobs:encode:failed:stream"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2 fi done diff --git a/bin/vgmetrics.sh b/bin/vgmetrics.sh new file mode 100644 index 0000000..5c43e6f --- /dev/null +++ b/bin/vgmetrics.sh @@ -0,0 +1,63 @@ +# Shared metric counters for the media scripts. Sourced, never executed. +# +# VG_MET_COMPONENT=sweep # a plain assignment, NOT a prefix on the next line: +# . /usr/local/bin/vgmetrics.sh && vgmet_init # bash discards "VAR=x . file" assignments on return +# vgmet_add "jobs:encode:done:stream" # +1 +# vgmet_add "seconds:encode:stream" 41.2 # +41.2 +# +# Counters accumulate across runs in MARK/metrics/.counters as plain "key value" lines, and a +# .pid file exists only while the script is executing. dumpd turns both into Prometheus text on +# its /metrics; nothing here renders or serves anything, so the scripts stay simple and there is one place +# that knows the metric names (bin/dumpd.py COUNTER_MAP). +# +# Keys are "name:label:label..." — the name picks the metric, the rest fill its labels in order. +# Writes are atomic (temp file + mv) and every failure is silent: metrics must never break a sweep. + +VG_MET_DIR="${VG_MET_DIR:-${MEDIA_ROOT:-/srv/media}/.markers/metrics}" +VG_MET_COMPONENT="${VG_MET_COMPONENT:-$(basename "${0%.sh}")}" +VG_MET_START=0 +declare -A VGMET=() + +vgmet_init() { + : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}" + : "${VG_MET_COMPONENT:=unknown}" + VG_MET_START=$(date +%s) + mkdir -p "$VG_MET_DIR" 2>/dev/null || return 0 + local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" k v + if [[ -r $f ]]; then + while read -r k v; do [[ -n ${k:-} && -n ${v:-} ]] && VGMET["$k"]="$v"; done < "$f" + fi + vgmet_add runs 1 + printf '%s\n' "$$" > "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null + trap vgmet_flush EXIT +} + +# vgmet_add [amount] — float-safe accumulate +vgmet_add() { + local k="$1" v="${2:-1}" + VGMET["$k"]=$(awk -v a="${VGMET[$k]:-0}" -v b="$v" 'BEGIN{ printf "%.3f", a+b }') +} + +vgmet_set() { VGMET["$1"]="$2"; } + +vgmet_flush() { + : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}" + : "${VG_MET_COMPONENT:=unknown}" + local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" tmp now k + rm -f "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null + [[ -d $VG_MET_DIR ]] || return 0 + now=$(date +%s) + vgmet_set last_finish "$now" + vgmet_set last_duration "$(( now - VG_MET_START ))" + tmp=$(mktemp "$VG_MET_DIR/.$VG_MET_COMPONENT.XXXXXX" 2>/dev/null) || return 0 + for k in "${!VGMET[@]}"; do printf '%s %s\n' "$k" "${VGMET[$k]}"; done > "$tmp" 2>/dev/null + chmod 0644 "$tmp" 2>/dev/null + mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" 2>/dev/null +} + +# start/elapsed for one job +vgmet_time() { date +%s.%N; } +vgmet_since() { awk -v s="$1" -v e="$(date +%s.%N)" 'BEGIN{ printf "%.3f", e-s }'; } + +# No-op fallbacks are defined by the callers if this file is missing, so a half-finished deploy cannot +# break normalising. Keep every function above failure-tolerant. diff --git a/check.sh b/check.sh index d8e069c..4f67d04 100755 --- a/check.sh +++ b/check.sh @@ -9,11 +9,11 @@ for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh register [[ -f $f ]] || continue bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; } done -for f in configure.py bin/dumpd.py bin/vgstrip.py; do +for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py observability/kibana-alarms.py; do python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; } done rm -rf __pycache__ bin/__pycache__ -python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } +python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } node -e " const fs=require('fs'); for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } @@ -27,6 +27,24 @@ c.write_mediamtx('media.example.com'); yaml.safe_load(open('mediamtx.yml')); c.w assert 'authMethod: http' in open('mediamtx.yml').read(); assert 'idp-h-usr' in open('copyparty.conf').read() " ) && say "configure.py generators" ok || { say "configure.py generators" "FAILED"; fail=1; } rm -rf "$T" +# dumpd GET /metrics: valid Prometheus text out of script counters + a fake media tree +T=$(mktemp -d); mkdir -p "$T/.markers/metrics" "$T/streams/u/Title/.live" +printf 'runs 2\njobs:encode:done:stream 1\nseconds:encode:stream 12.5\nlast_finish 1762210000\n' > "$T/.markers/metrics/sweep.counters" +touch -d '2020-01-01' "$T/streams/u/Title/.live/old.mp4" +if MEDIA_ROOT="$T" MTX_API_URL=http://127.0.0.1:1 python3 - <<'PYMET' >/dev/null 2>&1 +import re, sys +sys.path.insert(0, "bin") +import dumpd +text = dumpd.metrics_text() +for line in text.splitlines(): + assert line.startswith("#") or re.fullmatch(r'[a-z_]+(\{[^}]*\})? -?[0-9.]+', line), line +assert 'vanguard_media_jobs_total{component="sweep",kind="stream",outcome="done",verdict="encode"} 1' in text +assert 'vanguard_media_last_finish_timestamp_seconds{component="sweep"} 1762210000' in text # not 1.76e+09 +assert 'vanguard_media_pending{kind="stream"} 1' in text +assert 'vanguard_mediamtx_up 0' in text and 'vanguard_streams_active' not in text # no fake zeros +PYMET +then say "dumpd /metrics" ok; else say "dumpd /metrics" "FAILED"; fail=1; fi +rm -rf "$T" bin/__pycache__ if command -v caddy >/dev/null; then T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('media.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com','f.example.com','s.example.com','c.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; } rm -rf "$T" diff --git a/configure.py b/configure.py index 69d377d..38a6f0b 100755 --- a/configure.py +++ b/configure.py @@ -24,7 +24,8 @@ Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/va Discord bot token/guild), folders, restarts Set VG_DRY=1 to write files only (no sudo / docker / folders). """ -import getpass, hashlib, os, re, secrets, shutil, subprocess, sys +import base64, getpass, hashlib, json, os, re, secrets, shutil, subprocess, sys, time +import urllib.error, urllib.request from pathlib import Path HERE = Path(__file__).resolve().parent @@ -390,8 +391,23 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b env.setdefault("BOT_DEPLOY_AUTO", "1") # 0 = refuse bundles posted by CI; install by hand instead env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID") # the only .env keys the bot's CI may set env.setdefault("DIAG_REQUIRE_GROUP", "media-admins") # who may see logs, metrics and alerts - env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook for Kibana rules AND the deadman + env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook: vanguard-alerts.sh delivery AND the deadman env.setdefault("KIBANA_ENCRYPTION_KEY", secrets.token_hex(32)) # without a fixed key Kibana loses saved objects on restart + # Elasticsearch security is ON (observability/kibana.yml explains why, and why nobody sees a login). + # These are service credentials, never typed by a person: Discord remains the only human login. + def pw(): return "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32] + env.setdefault("ES_ELASTIC_PASSWORD", pw()) # the elastic superuser; bootstrapped on first secured start + env.setdefault("ES_KIBANA_PASSWORD", pw()) # kibana_system: Kibana's own internal account + env.setdefault("VG_BEATS_USER", "vanguard_beats") + env.setdefault("VG_BEATS_PASSWORD", pw()) # what Filebeat and Metricbeat ship as + env.setdefault("VG_DIAG_USER", "vanguard_diag") # who every Kibana visitor silently becomes + env.setdefault("VG_DIAG_PASSWORD", pw()) + # What that visitor may do. superuser keeps today's behaviour exactly (everyone past the gate is an admin). + # Narrowing this to "editor" is step one of the read-only tier in STATUS.md 9b. + env.setdefault("DIAG_KIBANA_ROLES", "superuser") + env.setdefault("ALERT_INDEX", "vanguard-alerts") # the alert bus: Kibana writes, vanguard-alerts.sh delivers + env.setdefault("LOG_RETENTION_DAYS", "30") # ILM delete phase for filebeat-*/metricbeat-* + env.setdefault("ALERT_RETENTION_DAYS", "90") # how long delivered alerts stay queryable env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot env.setdefault("DISCORD_GUILD_ID", "") env["WATCH_HOST"] = watch_host or "watch.example.com" @@ -421,7 +437,7 @@ rtspAddress: 127.0.0.1:8554 rtspTransports: [tcp] rtspEncryption: "no" moq: no -metrics: yes # Prometheus text format for whichever metrics store is chosen (LOGGING.md); loopback only +metrics: yes # Prometheus text format, scraped by Metricbeat (LOGGING.md); loopback only metricsAddress: :9998 pprof: no @@ -559,9 +575,10 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis e = read_env() Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600) Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600) - # 0640 root:65534 — the alertmanager image runs as nobody(65534) and the deadman runs as root. A webhook # URL is a credential for that channel, so it is not world-readable. - run(["sudo", "install", "-m", "0640", "-o", "root", "-g", "65534", "alert-webhook", "/etc/vanguard/alert-webhook"]) + # 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be + # group-readable by nobody(65534) for the Alertmanager container, which no longer exists. + run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"]) run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")): run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"]) @@ -577,13 +594,27 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis Path("99-vanguard-elasticsearch.conf").write_text("vm.max_map_count=262144\n") run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"]) run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state + run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"]) # script counters -> dumpd /metrics + # The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in + # /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md). + media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py", + "bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()] + if media: + run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"]) + if Path("bin/vgmetrics.sh").exists(): + run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")): run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"]) run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"]) run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"]) + # The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord. + if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")): + run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"]) + run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"]) + run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"]) run(["sudo", "systemctl", "restart", "dumpd"]) print("== containers") - dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"] + dc = compose_cmd() # --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how # the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them. run(dc + ["up", "-d", "--build", "--remove-orphans"]) # recreates only services whose definition changed @@ -594,10 +625,138 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis print(" run docker compose restart mediamtx between streams (cuts a live recording). Adding/removing") print(" streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.") +def compose_cmd(): + """docker compose, with sudo only if this user cannot reach the daemon.""" + if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0: + return ["docker", "compose"] + return ["sudo", "docker", "compose"] + +def es_security(dc): + """Create Elasticsearch's native users after the cluster is up. Idempotent; safe to re-run. + + Security is on, so three accounts have to exist before Kibana or the Beats can work: + kibana_system Kibana's internal account (built in; only its password needs setting) + VG_BEATS_USER what Filebeat and Metricbeat authenticate as + VG_DIAG_USER who every Kibana visitor silently becomes (kibana.yml's anonymous provider) + The elastic superuser itself is bootstrapped by the container from ELASTIC_PASSWORD, which only works + while the cluster has never had security on. If that did not take, this says exactly what to run rather + than guessing: a reset would desync the password the compose healthcheck uses. + """ + env = read_env() + elastic_pw = env.get("ES_ELASTIC_PASSWORD", "") + if not elastic_pw: + print(" elasticsearch: no ES_ELASTIC_PASSWORD in .env (write_env not run?) — skipped"); return + if DRY: + print(" (dry) elasticsearch: set kibana_system + create " + f"{env.get('VG_BEATS_USER')} and {env.get('VG_DIAG_USER')}"); return + + base = "http://127.0.0.1:9200" + def call(method, path, body=None): + request = urllib.request.Request(base + path, method=method, + data=None if body is None else json.dumps(body).encode()) + request.add_header("Content-Type", "application/json") + request.add_header("Authorization", "Basic " + base64.b64encode(f"elastic:{elastic_pw}".encode()).decode()) + with urllib.request.urlopen(request, timeout=20) as response: + raw = response.read() + return json.loads(raw) if raw else {} + + status = None + for _ in range(45): # the cluster can take a while on a box busy with x264 + try: + call("GET", "/_cluster/health"); status = "ok"; break + except urllib.error.HTTPError as error: + status = error.code + if error.code == 401: break # answering, but our password is not the one it bootstrapped + time.sleep(4) + except Exception: + time.sleep(4) + if status == 401: + print(" elasticsearch: REFUSED the password in .env. The cluster already had security enabled, so") + print(" ELASTIC_PASSWORD no longer bootstraps anything. Set it to the real one:") + print(" docker compose exec elasticsearch bin/elasticsearch-reset-password -u elastic -b -s") + print(" put that value in .env as ES_ELASTIC_PASSWORD, then re-run --apply.") + return + if status != "ok": + print(" elasticsearch: not answering yet — re-run ./configure.py --apply once it is up"); return + + try: + # If security is somehow still off, every call below 400s with "no handler found for uri". Check once + # and say so plainly, rather than reporting a confusing failure on the first password write. + call("POST", "/_security/user/_has_privileges", {"cluster": ["monitor"]}) + except urllib.error.HTTPError as error: + if error.code == 400: + print(" elasticsearch: the security API is absent — this cluster is still running with") + print(" xpack.security.enabled=false. Recreate it: docker compose up -d elasticsearch") + return + try: + call("POST", "/_security/user/kibana_system/_password", {"password": env["ES_KIBANA_PASSWORD"]}) + print(" elasticsearch: kibana_system password set") + for user, password, roles in ( + (env["VG_BEATS_USER"], env["VG_BEATS_PASSWORD"], ["superuser"]), + (env["VG_DIAG_USER"], env["VG_DIAG_PASSWORD"], + [r.strip() for r in env.get("DIAG_KIBANA_ROLES", "superuser").split(",") if r.strip()]), + ): + call("PUT", f"/_security/user/{user}", {"password": password, "roles": roles, + "full_name": "Vanguard service account"}) + print(f" elasticsearch: user {user} -> {', '.join(roles)}") + except urllib.error.HTTPError as error: + print(f" elasticsearch: user setup failed ({error.code}) — {error.read()[:200].decode(errors='replace')}") + return + + # The alert bus. Kibana's rules write here through the .index connector (the only outward connector a + # basic licence allows), bin/vanguard-alerts.sh delivers from here to Discord, and the deadman watches + # for the heartbeat document. Pre-created with an explicit mapping so "level" and "rule" are keywords + # and therefore aggregatable, rather than whatever dynamic mapping the first document implies. + alert_index = env.get("ALERT_INDEX", "vanguard-alerts") + try: + call("PUT", f"/{alert_index}", { + "settings": {"number_of_replicas": 0}, + "mappings": {"properties": { + "@timestamp": {"type": "date"}, + "kind": {"type": "keyword"}, + "rule": {"type": "keyword"}, + "level": {"type": "keyword"}, + "service": {"type": "keyword"}, + "message": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 1024}}}, + }}}) + print(f" elasticsearch: alert index {alert_index} created") + except urllib.error.HTTPError as error: + body = error.read().decode(errors="replace") + print(f" elasticsearch: alert index {alert_index} already there" + if "resource_already_exists" in body else f" elasticsearch: alert index failed — {body[:160]}") + + # Retention. The Beats create their own ILM policies with a rollover but NO delete phase, so indices + # grow until the disk watermark turns them read-only — which the deadman reports but cannot undo. Inject + # a delete phase instead of writing a whole policy, so whatever the Beats chose for rollover is kept. + keep_days = env.get("LOG_RETENTION_DAYS", "30") + for policy in ("filebeat", "metricbeat"): + try: + current = call("GET", f"/_ilm/policy/{policy}")[policy]["policy"] + except (urllib.error.HTTPError, KeyError): + print(f" elasticsearch: ILM policy {policy} not created yet — re-run --apply after the beat has started") + continue + phases = current.setdefault("phases", {}) + if phases.get("delete", {}).get("min_age") == f"{keep_days}d": + print(f" elasticsearch: {policy} already deletes after {keep_days}d") + continue + phases["delete"] = {"min_age": f"{keep_days}d", "actions": {"delete": {}}} + call("PUT", f"/_ilm/policy/{policy}", {"policy": current}) + print(f" elasticsearch: {policy} indices now deleted after {keep_days}d") + + # The deadman runs on the host as root and must authenticate too. netrc, not -u, so the password never + # shows up in anyone's process list. + Path("es-netrc").write_text(f"machine 127.0.0.1 login elastic password {elastic_pw}\n") + run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "es-netrc", "/etc/vanguard/es-netrc"]) + Path("es-netrc").unlink(missing_ok=True) + # Kibana and the Beats were started before those accounts existed, so they are sitting in a retry loop. + run(dc + ["restart", "kibana", "filebeat", "metricbeat"]) + print(" elasticsearch: netrc written for the deadman; kibana + beats restarted") + def post_start_fixups(dc): """Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin.""" env = read_env() print("== post-start fixups (idempotent)") + es_security(dc) # before anything else: Kibana and the Beats cannot authenticate until this runs # Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it cfg = Path("/srv/media/.syncthing/config.xml") if not DRY and cfg.exists(): @@ -627,6 +786,32 @@ def post_start_fixups(dc): print(f" gitea: admin user {env['GITEA_ADMIN_USER']} created (password in .env)") else: print(" gitea: ok" if r.returncode == 0 else " gitea: not ready (install page or db?) — run ./configure.py --apply again later") else: print(" (dry) gitea admin user") + # Kibana first-run state: the alert connector, the heartbeat rule and the data view. Without this a + # fresh install has no way to deliver an alert and the deadman alarms about its own missing heartbeat. + # It needs an API key because Basic is not an accepted scheme for this Kibana (anonymous is the only + # enabled provider), and it waits for Kibana itself, which es_security restarted a moment ago. + setup = Path("observability/kibana-setup.py") + if setup.exists() and env.get("ES_ELASTIC_PASSWORD"): + if DRY: + print(" (dry) kibana: connector, heartbeat rule, data view") + else: + try: + request = urllib.request.Request( + "http://127.0.0.1:9200/_security/api_key", method="POST", + data=json.dumps({"name": "vanguard-setup", "expiration": "10m"}).encode()) + request.add_header("Content-Type", "application/json") + request.add_header("Authorization", "Basic " + base64.b64encode( + f"elastic:{env['ES_ELASTIC_PASSWORD']}".encode()).decode()) + with urllib.request.urlopen(request, timeout=20) as response: + api_key = json.load(response)["encoded"] + result = subprocess.run([sys.executable, str(setup)], text=True, capture_output=True, + env={**os.environ, "KEY": api_key, + "ALERT_INDEX": env.get("ALERT_INDEX", "vanguard-alerts")}) + for line in (result.stdout + result.stderr).strip().splitlines(): + print(" kibana:", line) + except Exception as error: + print(f" kibana: first-run setup skipped ({error}) — run it by hand, see its docstring") + # Jellyfin plugin from the committed zip if Path("installJellyfinPlugin.sh").exists(): if DRY: print(" (dry) installJellyfinPlugin.sh") @@ -641,8 +826,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_ if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}") print(f"== live : https://{live_host}/ (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live : (not set — run --full)") alert_hook = read_env().get("ALERT_WEBHOOK_URL", "") - print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); Kibana rules and the deadman post there" if alert_hook - else "NO WEBHOOK — Kibana has nowhere to send and the deadman only writes to the journal. Set ALERT_WEBHOOK_URL in .env.")) + print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); vanguard-alerts.timer and the deadman post there" if alert_hook + else "NO WEBHOOK — Kibana still records alerts in the vanguard-alerts index, but nothing delivers them: " + "vanguard-alerts.sh and the deadman write to the journal and exit non-zero. Set ALERT_WEBHOOK_URL in .env.")) print(f"== diag : https://{diag_host}/ (Kibana: logs, dashboards, alerting; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)") print(f"== bot web : https://{bot_host}/ (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web : (not set — run --full)") print(f"== gitea : https://{git_host}/ ssh port 2222 admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')} (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea : (not set — run --full)") @@ -765,6 +951,11 @@ def main(): streamers, accounts = read_streamers(), read_accounts() admin_pw = accounts.pop("admin", "") editors = accounts + if mode == "--es-security": + # Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an + # --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but + # publishes nothing — three confusing symptoms with one cause. + es_security(compose_cmd()); return if mode == "--show": show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return if mode == "--apply": diff --git a/diag/index.html b/diag/index.html deleted file mode 100644 index 03c1f9f..0000000 --- a/diag/index.html +++ /dev/null @@ -1,37 +0,0 @@ - - -Vanguard diagnostics - -
-

Vanguard diagnostics

-

Logs, metrics and alerts for this server. Visible to media-admins.

- -Logsevery container and our own timers, last 30 days · VictoriaLogs -Metricshost, containers, MediaMTX, Caddy, last 90 days · VictoriaMetrics -Alertswhat is firing now, and silences · Alertmanager -Metric rulesdisk, load, memory, RAID, container restart loops -Log rulessweep failures, publish denials, failed bot releases - -
A deadman runs on the host, outside all of this. Every five minutes it checks that each -service here still answers, that the alerting path is still delivering, and that the disks are not filling, and -it reports to Discord itself. Nothing on this page can tell you it has died, which is the point.
- - -
diff --git a/docker-compose.yml b/docker-compose.yml index 21c1be4..a5bebf0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -72,7 +72,6 @@ services: - ./dock:/srv/dock:ro - ./clips:/srv/clips:ro # clip-intake self-service page - ./about:/srv/about:ro # gateway page (/about) - - ./diag:/srv/diag:ro # diagnostics index (diag.HOST) - /srv/live:/srv/live:ro # live portal static build (deployLive.sh) - caddy_data:/data - caddy_config:/config @@ -235,6 +234,17 @@ services: - ${BOT_APP_DIR:-/srv/bot/app}:/app # CI bundle installed by ./deployBotBundle.sh (code + vendor/) - /srv/media/clips:/media/clips # rw: for a clip-submissions feature (writes into clips//) - /srv/appdata/discord-bot:/data # persistent scratch (sqlite, caches) if the bot wants files + # Asks the BOT about itself rather than just proving the port accepts a connection: /health answers 503 + # unless the Discord gateway is connected AND the database responds, and -f turns that into an unhealthy + # container, which Metricbeat's docker healthcheck metricset turns into an alarm. Requires the bot release + # that made /health unauthenticated and status-bearing — on older code every check 401s and -f fails, + # so ship the bot first. start_period covers command sync and the first roster load. + healthcheck: + test: ["CMD-SHELL", "curl -fsS -m 5 http://127.0.0.1:8000/health >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s extra_hosts: - "host.docker.internal:host-gateway" networks: @@ -330,7 +340,18 @@ services: # machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics. ES_JAVA_OPTS: -Xms2g -Xmx2g discovery.type: single-node - xpack.security.enabled: "false" # loopback only, behind Caddy + Authentik; no second login + # ON. Kibana's Observability apps (Infrastructure/Metrics) ask Elasticsearch whether the current user + # holds a privilege; with security off that API does not exist and they fail on every view. Humans still + # never see a password: Kibana signs everyone in as one user (observability/kibana.yml). Caddy + + # Authentik + DIAG_REQUIRE_GROUP remains the only real gate. + xpack.security.enabled: "true" + xpack.security.http.ssl.enabled: "false" # plain HTTP, published on loopback only + xpack.security.transport.ssl.enabled: "false" # single-node: no transport traffic, and the check is skipped + xpack.security.enrollment.enabled: "false" + xpack.security.autoconfiguration.enabled: "false" + # Bootstraps the elastic superuser on first start with security on. configure.py generates it and then + # sets up kibana_system and the service users; the error below means .env predates that. + ELASTIC_PASSWORD: "${ES_ELASTIC_PASSWORD:?run ./configure.py --apply first — it generates the Elastic passwords}" xpack.ml.enabled: "false" # not used, and it reserves memory cluster.routing.allocation.disk.watermark.low: 85% cluster.routing.allocation.disk.watermark.high: 90% @@ -344,7 +365,7 @@ services: volumes: - /srv/appdata/elasticsearch:/usr/share/elasticsearch/data healthcheck: - test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"] + test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"] interval: 20s timeout: 10s retries: 12 @@ -356,14 +377,16 @@ services: labels: { vanguard.service: kibana, vanguard.category: infra } image: docker.elastic.co/kibana/kibana:9.2.0 container_name: kibana + # Everything non-secret lives in observability/kibana.yml (which explains the auth model); only the + # generated secrets arrive as environment variables, which that file interpolates. environment: - ELASTICSEARCH_HOSTS: http://elasticsearch:9200 - SERVER_PUBLICBASEURL: https://${DIAG_HOST:-diag.example.com} - SERVER_BASEPATH: "" # served at the root of diag.HOST, so no prefix rewriting to get wrong - TELEMETRY_OPTIN: "false" - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY} - XPACK_REPORTING_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY} - XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY} + DIAG_HOST: ${DIAG_HOST:-diag.example.com} + KIBANA_ENCRYPTION_KEY: ${KIBANA_ENCRYPTION_KEY} + ES_KIBANA_PASSWORD: "${ES_KIBANA_PASSWORD:?run ./configure.py --apply first}" + VG_DIAG_USER: ${VG_DIAG_USER:-vanguard_diag} + VG_DIAG_PASSWORD: "${VG_DIAG_PASSWORD:?run ./configure.py --apply first}" + volumes: + - ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro ports: - "127.0.0.1:5601:5601" depends_on: @@ -375,6 +398,9 @@ services: labels: { vanguard.service: filebeat, vanguard.category: infra } image: docker.elastic.co/beats/filebeat:9.2.0 container_name: filebeat + environment: # Elasticsearch requires auth now; the config interpolates these + ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats} + ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}" user: root command: ["filebeat", "-e", "--strict.perms=false"] volumes: @@ -382,6 +408,7 @@ services: - /var/lib/docker/containers:/var/lib/docker/containers:ro - /var/run/docker.sock:/var/run/docker.sock # autodiscover reads container names and our vanguard.* labels - /var/log/journal:/var/log/journal:ro # our own systemd units + - /run/log/journal:/run/log/journal:ro # ...and the volatile journal, if this host has no persistent one - /etc/machine-id:/etc/machine-id:ro - filebeat_data:/usr/share/filebeat/data depends_on: @@ -393,6 +420,9 @@ services: labels: { vanguard.service: metricbeat, vanguard.category: infra } image: docker.elastic.co/beats/metricbeat:9.2.0 container_name: metricbeat + environment: # Elasticsearch requires auth now; the config interpolates these + ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats} + ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}" user: root command: ["metricbeat", "-e", "--strict.perms=false"] # system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards. diff --git a/install.sh b/install.sh index 136a57b..0abc876 100755 --- a/install.sh +++ b/install.sh @@ -38,6 +38,7 @@ echo "== media user + folders" getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media usermod -aG docker "$OWNER" 2>/dev/null || true install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing} +install -d -o 1000 -g 1000 /srv/media/.markers/metrics # script counters -> dumpd /metrics install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs} install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner @@ -56,19 +57,21 @@ done echo "== host scripts + units" install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/ +install -m 0644 bin/vgmetrics.sh /usr/local/bin/ # sourced by the sweep/vgdeband; counters for dumpd's /metrics install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it -# Compose bind-mounts this as a FILE; without it Docker would create a directory and alertmanager would not -# start. configure.py rewrites it from ALERT_WEBHOOK_URL on every apply; an empty one means "log only". +install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord +# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it +# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero". if [[ ! -f /etc/vanguard/alert-webhook ]]; then install -d -m 0750 /etc/vanguard - install -m 0640 -o root -g 65534 /dev/null /etc/vanguard/alert-webhook + install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook # root timers are the only readers fi install -d -m 0750 /etc/vanguard install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/ systemctl daemon-reload -systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer +systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer echo "== sysctl" cat > /etc/sysctl.d/90-vanguard.conf <<'SYS' diff --git a/observability/filebeat.yml b/observability/filebeat.yml index d1bf9ce..43d96b9 100644 --- a/observability/filebeat.yml +++ b/observability/filebeat.yml @@ -2,6 +2,11 @@ # Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service / # vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana # needs no parsing. See LOGGING.md for what each service's lines look like. +# +# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and +# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the +# process up with zero harvesters. Container logs are read by filestream + the container parser, and every +# filestream input needs its own stable id. filebeat.autodiscover: providers: @@ -10,8 +15,20 @@ filebeat.autodiscover: templates: - condition: ~ # every container config: - - type: container + - type: filestream + id: container-${data.docker.container.id} paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"] + parsers: + - container: ~ + prospector.scanner.symlinks: true + # Stamp the id here rather than letting add_docker_metadata guess it from the path. + processors: + - add_fields: + target: container + fields: + id: "${data.docker.container.id}" + name: "${data.docker.container.name}" + image.name: "${data.docker.container.image}" filebeat.inputs: - type: journald @@ -25,7 +42,8 @@ filebeat.inputs: - "_SYSTEMD_UNIT=vanguard-deadman.service" processors: - - add_docker_metadata: ~ + - add_docker_metadata: + match_fields: ["container.id"] # the id the autodiscover template stamped above - add_host_metadata: ~ # promote our labels to real fields so Kibana filters on service.name and vanguard.category - copy_fields: @@ -36,15 +54,36 @@ processors: to: vanguard.category fail_on_error: false ignore_missing: true + # The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in + # a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's + # vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target + # that already exists, so the first rule that matches wins and the other is skipped. + - copy_fields: + fields: + - from: container.labels.vanguard_service + to: event.dataset + - from: systemd.unit + to: event.dataset + fail_on_error: false + ignore_missing: true - drop_fields: - fields: ["agent.ephemeral_id", "ecs.version", "input.type", "log.offset"] + fields: ["agent.ephemeral_id", "ecs.version", "log.offset"] ignore_missing: true output.elasticsearch: hosts: ["http://elasticsearch:9200"] + # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These + # come from the container environment, which compose fills from .env — no secret is committed here. + username: "${ES_BEATS_USER}" + password: "${ES_BEATS_PASSWORD}" -setup.kibana: +setup.kibana: # no credentials: Kibana's anonymous provider signs these in host: "http://kibana:5601" setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h) +# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the +# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround. +setup.template.overwrite: true +setup.template.settings: + index.number_of_replicas: 0 logging.level: info diff --git a/observability/kibana-alarms.py b/observability/kibana-alarms.py new file mode 100755 index 0000000..efa853b --- /dev/null +++ b/observability/kibana-alarms.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +"""The real alarm rules. Idempotent: creates what is missing, leaves what exists. + +Every rule is an Elasticsearch query rule that counts matching documents in a window and writes one document +into the alert bus through the "Vanguard alert index" connector, which bin/vanguard-alerts.sh delivers to +Discord. Run observability/kibana-setup.py first — it creates that connector. + +FIRING CADENCE: notify_when=onThrottleInterval with throttle=12h, so an alarm that stays true writes again +every 12 hours and stops when the condition clears. The delivery script's own 15-minute per-rule suppression +sits well inside that and never interferes. + +WHAT IS DELIBERATELY NOT HERE +Elasticsearch, Kibana and Beat liveness, and the disks, are bin/vanguard-deadman.sh's job. Nothing that +depends on Kibana being alive can report that Kibana is dead, so those checks stay outside the stack and +duplicating them here would be worse than useless: they would go quiet in exactly the case that matters. + + pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-) + KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \ + -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])') + KEY="$KEY" python3 observability/kibana-alarms.py + +To change a rule, edit it in Kibana or delete it there and re-run this. Thresholds here are the defaults: + FREE_PCT 15 alarm when a real filesystem or memory has less than this percent free + STUCK_SECONDS 3600 alarm when a finished recording has waited this long to be normalised + THROTTLE 12h how often an unresolved alarm repeats +""" +import json, os, urllib.error, urllib.request + +KIBANA = "http://127.0.0.1:5601" +HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1", + "Authorization": "ApiKey " + os.environ["KEY"]} +CONNECTOR_NAME = "Vanguard alert index" + +FREE_PCT = float(os.environ.get("FREE_PCT", "15")) +STUCK_SECONDS = int(os.environ.get("STUCK_SECONDS", "3600")) +THROTTLE = os.environ.get("THROTTLE", "12h") +USED_MAX = round(1 - FREE_PCT / 100, 4) # 15% free -> used.pct >= 0.85 + +# Filesystems that are not disks. Without this, every tmpfs and overlay mount reports ~100% used and the +# disk alarm would fire permanently on something that cannot fill up. +PSEUDO_FS = ["tmpfs", "devtmpfs", "overlay", "squashfs", "ramfs", "autofs", "nsfs", "tracefs", "debugfs", + "configfs", "mqueue", "hugetlbfs", "pstore", "bpf", "devpts", "proc", "sysfs", "cgroup", + "cgroup2", "fusectl", "securityfs", "efivarfs", "binfmt_misc", "fuse.lxcfs", "iso9660"] + +def rule(name, level, why, query, *, index="metricbeat-*", window=10, comparator=">", threshold=0): + return {"name": name, "level": level, "why": why, "query": query, + "index": index, "window": window, "comparator": comparator, "threshold": threshold} + +ALARMS = [ + rule("Disk free below %g%%" % FREE_PCT, "error", + "A filesystem is nearly full. Elasticsearch turns its indices READ-ONLY at the flood watermark and " + "does not undo that by itself, and recordings stop being published.", + {"query": {"bool": { + "filter": [{"range": {"system.filesystem.used.pct": {"gte": USED_MAX}}}], + "must_not": [{"terms": {"system.filesystem.type": PSEUDO_FS}}]}}}), + + rule("Memory free below %g%%" % FREE_PCT, "warn", + "Memory is nearly exhausted. Elasticsearch has a pinned 2 GB heap, so the pressure is usually " + "ffmpeg or Jellyfin, and the kernel will start killing processes rather than queueing.", + {"query": {"range": {"system.memory.actual.used.pct": {"gte": USED_MAX}}}}), + + rule("Discord bot not running", "error", + "No metrics from the discord-bot container for 5 minutes: it is stopped, removed, or restarting too " + "fast to be sampled. Nothing it does is happening.", + {"query": {"term": {"container.name": "discord-bot"}}}, + window=5, comparator="<", threshold=1), + + rule("Discord bot unhealthy", "error", + "The bot is running but reports itself degraded: its /health says the Discord gateway is " + "disconnected or the database does not answer. It is up and doing nothing.", + {"query": {"bool": {"filter": [{"term": {"container.name": "discord-bot"}}, + {"term": {"docker.healthcheck.status": "unhealthy"}}]}}}, + window=5), + + rule("Recordings stuck waiting to be normalised", "warn", + "A finished recording has waited over %d minutes in .live/. It is not in Jellyfin, not in copyparty " + "and not announced, so to everyone else it does not exist." % (STUCK_SECONDS // 60), + {"query": {"range": {"prometheus.vanguard_media_pending_oldest_seconds.value": {"gt": STUCK_SECONDS}}}}), + + rule("Normalise failures parked", "warn", + "One or more files have a normalize-failed log, so the sweep skips them forever until it is deleted. " + "The log now contains ffmpeg's stderr, which says why.", + {"query": {"range": {"prometheus.vanguard_media_failed_files.value": {"gt": 0}}}}), +] + +def req(method, path, body=None): + request = urllib.request.Request(KIBANA + path, method=method, + data=None if body is None else json.dumps(body).encode()) + for key, value in HEADERS.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + raw = response.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as error: + print(f" {method} {path} -> HTTP {error.code}") + print(" " + error.read().decode(errors="replace")[:600]) + raise SystemExit(1) + +connectors = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")} +if CONNECTOR_NAME not in connectors: + raise SystemExit(f"connector {CONNECTOR_NAME!r} is missing — run observability/kibana-setup.py first") +connector_id = connectors[CONNECTOR_NAME] + +present = {r["name"] for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])} +created = skipped = 0 +for alarm in ALARMS: + if alarm["name"] in present: + print("already present:", alarm["name"]) + skipped += 1 + continue + made = req("POST", "/api/alerting/rule", { + "name": alarm["name"], + "rule_type_id": ".es-query", + "consumer": "stackAlerts", + "enabled": True, + "schedule": {"interval": "1m"}, + "params": { + "searchType": "esQuery", + "index": [alarm["index"]], + "timeField": "@timestamp", + "esQuery": json.dumps(alarm["query"]), + "size": 1, + "aggType": "count", + "groupBy": "all", + "thresholdComparator": alarm["comparator"], + "threshold": [alarm["threshold"]], + "timeWindowSize": alarm["window"], + "timeWindowUnit": "m", + "excludeHitsFromPreviousRun": False, + }, + "actions": [{ + "group": "query matched", + "id": connector_id, + "params": {"documents": [{ + "kind": "alarm", + "rule": "{{rule.name}}", + "level": alarm["level"], + "message": alarm["why"] + " [{{context.message}}]", + }]}, + # every 12 h for as long as it stays true, then silence when it clears + "frequency": {"summary": False, "notify_when": "onThrottleInterval", "throttle": THROTTLE}, + }], + }) + print("created:", made["name"], "|", alarm["level"], "| repeats every", THROTTLE) + created += 1 + +print(f"\n{created} created, {skipped} already there. Disk/memory threshold {FREE_PCT:g}% free; " + f"stuck after {STUCK_SECONDS}s.") diff --git a/observability/kibana-setup.py b/observability/kibana-setup.py new file mode 100755 index 0000000..3e0af4e --- /dev/null +++ b/observability/kibana-setup.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Kibana first-run state. Idempotent; safe to re-run. configure.py --apply runs this for you. + + connector "Vanguard alert index" — the .index connector, writing into the vanguard-alerts index + rule "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc + data view "Vanguard logs + metrics" — filebeat-*,metricbeat-* so Discover and Lens have one source + +WHY AN INDEX AND NOT A WEBHOOK +On a basic license the only connector types Kibana allows are .index and .server-log; every connector that +leaves the stack (webhook, Slack, email, PagerDuty…) is Gold or above and returns + 403 Action type .webhook is disabled because your basic license does not support it +So Kibana cannot talk to Discord. Instead every rule writes a document into ONE index, which is the alert +bus, and bin/vanguard-alerts.sh (a host timer) delivers from there to Discord and later to the bot. That is +strictly simpler than the webhook route: no Docker-bridge hop, no dumpd endpoint, and alerts are queryable +and replayable after the fact because they are just documents. + +THE HEARTBEAT +bin/vanguard-deadman.sh alarms when no heartbeat document has arrived recently. It is the only check that +catches the ALERTING pipeline itself dying — if Kibana, its rule engine or the connector stops, no ordinary +alert can report it, because the thing that would report it is what broke. The rule queries metricbeat-*, +which always has recent documents, so it doubles as a Metricbeat liveness check. + +Run it with an Elasticsearch API key: Basic is not an accepted scheme for this Kibana (the anonymous provider +is the only one enabled, so only ApiKey and Bearer are taken). + + pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-) + KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \ + -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])') + KEY="$KEY" python3 observability/kibana-setup.py + +Order matters: run this AFTER security is on. A rule created while security was off holds no API key and +stops working the moment it is enabled, until it is disabled and re-enabled. + +ADDING A REAL ALARM: author it in Kibana as usual, point its action at the "Vanguard alert index" connector, +and give the document a message plus a level (warn|error). Do NOT set kind=heartbeat — that value is how the +delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle +for real alarms; only the heartbeat wants to fire every interval. +""" +import json, os, time, urllib.error, urllib.request + +KIBANA = "http://127.0.0.1:5601" +ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts") +HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1", + "Authorization": "ApiKey " + os.environ["KEY"]} + +def wait_for_kibana(seconds=300): + """Kibana has to be *available*, not merely listening. + + configure.py --es-security restarts Kibana as its last step, so anything that calls the Kibana API + immediately afterwards races its boot and gets a connection reset. Kibana also answers /api/status while + still starting, so the status level is what has to be checked, not the port. + """ + deadline = time.time() + seconds + announced = False + while time.time() < deadline: + try: + with urllib.request.urlopen(KIBANA + "/api/status", timeout=10) as response: + level = json.load(response).get("status", {}).get("overall", {}).get("level") + if level == "available": + print("kibana: available") + return + state = level or "no level" + except Exception as error: + state = type(error).__name__ + if not announced: + print(f"kibana: waiting for it to become available ({state})…") + announced = True + time.sleep(5) + raise SystemExit(f"kibana did not become available within {seconds}s — nothing was created") + +def req(method, path, body=None): + request = urllib.request.Request(KIBANA + path, method=method, + data=None if body is None else json.dumps(body).encode()) + for key, value in HEADERS.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + raw = response.read() + return json.loads(raw) if raw else {} + except urllib.error.HTTPError as error: + print(f" {method} {path} -> HTTP {error.code}") + print(" " + error.read().decode(errors="replace")[:600]) + raise SystemExit(1) + +wait_for_kibana() + +# 1. the alert bus connector +CONNECTOR_NAME = "Vanguard alert index" +existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")} +if CONNECTOR_NAME in existing: + connector_id = existing[CONNECTOR_NAME] + print("connector already present:", connector_id) +else: + connector_id = req("POST", "/api/actions/connector", { + "name": CONNECTOR_NAME, + "connector_type_id": ".index", + # executionTimeField stamps every document, so a rule body never has to get the timestamp right. + "config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"}, + "secrets": {}})["id"] + print("connector created:", connector_id, "->", ALERT_INDEX) + +# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked +# harmless and meant the data view was never created on any system where the rule already existed, which +# is every re-run. +RULE_NAME = "Vanguard alerting heartbeat" +existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", []) + if r["name"] == RULE_NAME), None) +if existing_rule: + print("rule already present:", existing_rule["id"], + "enabled" if existing_rule.get("enabled") else "DISABLED") +else: + rule = req("POST", "/api/alerting/rule", { + "name": RULE_NAME, + "rule_type_id": ".es-query", + "consumer": "stackAlerts", + "enabled": True, + "schedule": {"interval": "1m"}, + "params": { + "searchType": "esQuery", + "index": ["metricbeat-*"], + "timeField": "@timestamp", + "esQuery": json.dumps({"query": {"match_all": {}}}), + "size": 1, + "aggType": "count", + "groupBy": "all", + "thresholdComparator": ">", + "threshold": [0], + "timeWindowSize": 5, + "timeWindowUnit": "m", + # re-evaluate the whole window every run, so the rule stays active instead of going quiet once + # it has already seen those documents + "excludeHitsFromPreviousRun": False, + }, + # onActiveAlert writes on EVERY check. Kibana's default only acts when a rule CHANGES status, which + # would produce one heartbeat and then silence — the deadman would read that as a dead pipeline. + "actions": [{ + "group": "query matched", + "id": connector_id, + "params": {"documents": [{ + "kind": "heartbeat", # the delivery script drops these; never reach Discord + "rule": "{{rule.name}}", + "level": "info", + "message": "kibana rule engine and the index connector are alive", + }]}, + "frequency": {"summary": False, "notify_when": "onActiveAlert", "throttle": None}, + }], + }) + print("rule created:", rule["id"], "| enabled:", rule.get("enabled")) + +# 3. the data view. Without one, Discover and Lens have nothing to point at and every panel starts with a +# detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and +# metrics, which is the whole reason they share a cluster. +VIEW_TITLE = "filebeat-*,metricbeat-*" +views = req("GET", "/api/data_views").get("data_view", []) +if any(v.get("title") == VIEW_TITLE for v in views): + print("data view already present:", VIEW_TITLE) +else: + view = req("POST", "/api/data_views/data_view", { + "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}}) + print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE) diff --git a/observability/kibana.yml b/observability/kibana.yml new file mode 100644 index 0000000..06388e0 --- /dev/null +++ b/observability/kibana.yml @@ -0,0 +1,41 @@ +# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates +# into .env, so this file is committed and contains none. +# +# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN +# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch +# whether the current user holds a privilege. With security off that API does not exist, and the +# Infrastructure / Metrics app fails on every page view with +# no handler found for uri [/_security/user/_has_privileges] +# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow: +# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as +# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik +# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host. +# +# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same +# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and +# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey +# scheme and have Caddy inject a key chosen by Authentik group). + +server.host: "0.0.0.0" +server.publicBaseUrl: "https://${DIAG_HOST}" +# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty: +# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to +# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values. +telemetry.optIn: false + +elasticsearch.hosts: ["http://elasticsearch:9200"] +elasticsearch.username: "kibana_system" # Kibana's own internal account, NOT what visitors become +elasticsearch.password: "${ES_KIBANA_PASSWORD}" + +# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules). +xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}" +xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}" +xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}" + +# The only provider, so there is no login page and no login selector: every request is already signed in. +xpack.security.authc.providers: + anonymous.anonymous1: + order: 0 + credentials: + username: "${VG_DIAG_USER}" + password: "${VG_DIAG_PASSWORD}" diff --git a/observability/metricbeat.yml b/observability/metricbeat.yml index e87eef5..ac0e2e9 100644 --- a/observability/metricbeat.yml +++ b/observability/metricbeat.yml @@ -27,6 +27,15 @@ metricbeat.modules: metrics_path: /metrics hosts: ["http://127.0.0.1:2019"] + - module: prometheus # dumpd: OUR metrics — normalise queue, encode time, live streams + period: 30s + metrics_path: /metrics + hosts: ["http://127.0.0.1:8080"] # dumpd's loopback listener; Caddy routes no host to /metrics + # These are real counters, so let them arrive typed: Kibana then gets .rate (per-second) as well as the + # raw total, which is what makes "encode seconds per hour" a chart rather than a staircase. + use_types: true + rate_counters: true + processors: - add_host_metadata: ~ # The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a @@ -43,9 +52,18 @@ processors: output.elasticsearch: hosts: ["http://127.0.0.1:9200"] + # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These + # come from the container environment, which compose fills from .env — no secret is committed here. + username: "${ES_BEATS_USER}" + password: "${ES_BEATS_PASSWORD}" -setup.kibana: +setup.kibana: # no credentials: Kibana's anonymous provider signs these in host: "http://127.0.0.1:5601" setup.dashboards.enabled: true # the system and docker dashboards, so there is something to look at +# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the +# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround. +setup.template.overwrite: true +setup.template.settings: + index.number_of_replicas: 0 logging.level: info diff --git a/systemd/vanguard-alerts.service b/systemd/vanguard-alerts.service new file mode 100644 index 0000000..5381052 --- /dev/null +++ b/systemd/vanguard-alerts.service @@ -0,0 +1,9 @@ +[Unit] +Description=Vanguard alerts: deliver Kibana's alert documents to Discord +After=docker.service + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/vanguard-alerts.sh +# Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts +# index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md. diff --git a/systemd/vanguard-alerts.timer b/systemd/vanguard-alerts.timer new file mode 100644 index 0000000..d23c182 --- /dev/null +++ b/systemd/vanguard-alerts.timer @@ -0,0 +1,10 @@ +[Unit] +Description=Deliver Vanguard alerts every minute + +[Timer] +OnBootSec=3min +OnUnitActiveSec=1min +AccuracySec=15s + +[Install] +WantedBy=timers.target