techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Browse Source 
Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 19:58:06 -06:00
parent ef357c6d82
commit b38d40ab0c
20 changed files with 1322 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
LOGGING.md
View File

@@ -1,8 +1,10 @@
# Logging framework
Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI
at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so
adopting it is a collector config rather than a code change.
Decided 2026-11-01. The store was chosen, built and replaced on 2026-11-03: a VictoriaLogs stack went in
first and was torn out the same day for producing no dashboards, so the store is now **Elasticsearch with
Kibana**, served at the root of the `diag.BASE` host (Phase 13 in `CommunityDiscordMedia.md`). Logs arrive via
Filebeat, metrics via Metricbeat, into the same cluster so one dashboard can carry both. Everything upstream
of the store was fixed first, so the swap was a collector config rather than a code change.
## Where the lines come from (the only two taps a collector needs)
@@ -11,21 +13,66 @@ adopting it is a collector config rather than a code change.
| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback.
Filebeat reads both: a docker autodiscover template per container (filestream + the container parser) and a
journald input for our units. Metrics are the same story: see the taps table below, all on loopback.
## Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata)
## Metrics taps (Prometheus text format, loopback only — Metricbeat scrapes them)
| endpoint | what | from |
|---|---|---|
| `http://127.0.0.1:9100/metrics` | host: CPU, RAM, disk space + I/O, network, load, md RAID | node_exporter (host network, pid host) |
| `http://127.0.0.1:9101/metrics` | per container: CPU, memory, network, block I/O — labelled with `vanguard.service` / `vanguard.category` | cAdvisor |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet:
metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against
VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on
`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
Host and per-container metrics (CPU, RAM, disk, network, block I/O, restarts) come from Metricbeat's `system`
and `docker` modules, not from an exporter — node_exporter and cAdvisor were removed when the Elastic stack
went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
### Our own metrics, and how to add one
MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
keeping up", because the sweep is a shell script on a timer. So the scripts keep counters and dumpd renders
them.
- `bin/vgmetrics.sh` is sourced by `normalize-sweep.sh` and `vgdeband.sh`. It accumulates `key value` lines in
`/srv/media/.markers/metrics/<component>.counters` and keeps a `<component>.pid` file for as long as the
script is executing. It renders nothing and serves nothing, so the scripts stay simple.
- `bin/dumpd.py` turns those files into Prometheus text on `GET /metrics`, adds gauges it can compute at
scrape time, and asks the MediaMTX API for the live path list. `COUNTER_MAP` there is the single place that
knows the metric names.
| metric | type | what it answers |
|---|---|---|
| `vanguard_media_pending{kind}` | gauge | finished recordings waiting to be normalised. **The capacity signal**: if this trends up across a night, CPU encoding is not keeping pace |
| `vanguard_media_pending_oldest_seconds` | gauge | how long the oldest one has waited |
| `vanguard_media_recording` | gauge | recordings MediaMTX is still writing (age under `MINS_IDLE`) |
| `vanguard_media_encoding_in_flight` | gauge | ffmpeg output files (`*.part`) open right now |
| `vanguard_media_running{component}` | gauge | 1 while the sweep or vgdeband is executing, by live pid, not a stale marker |
| `vanguard_media_jobs_total{component,verdict,outcome,kind}` | counter | remuxes and encodes finished, split by success and by stream vs clip |
| `vanguard_media_job_seconds_total{component,verdict,kind}` | counter | seconds spent in ffmpeg. Divide by the job count for mean encode time |
| `vanguard_media_runs_total{component}` | counter | script starts. Flat means the timer stopped |
| `vanguard_media_deband_total{component}` | counter | timecode bands cropped off |
| `vanguard_media_last_finish_timestamp_seconds{component}` | gauge | when it last finished |
| `vanguard_media_last_run_seconds{component}` | gauge | how long that run took |
| `vanguard_media_failed_files` | gauge | files parked with a `normalize-failed.log` |
| `vanguard_mediamtx_up` | gauge | did the MediaMTX API answer the scrape |
| `vanguard_streams_active` | gauge | streams publishing now, derived `.webrtc` mirrors excluded |
| `vanguard_streams_mirrors` | gauge | those mirrors, counted separately |
| `vanguard_stream_readers` | gauge | viewers attached across all paths |
To add a counter: pick a key of the form `name:label:label`, call `vgmet_add` with it, and add `name` to
`COUNTER_MAP` with the label names in the same order. Nothing else changes, and a key dumpd does not
recognise is ignored rather than breaking the scrape.
Three rules this design exists to keep. Metrics must never break a sweep, so every write is atomic and every
failure in the helper is silent, and a missing helper degrades to no-ops. A scrape must never disturb the
stack, so dumpd never takes the sweep's flock and the filesystem globs are depth-limited rather than walking
the media tree. And the counters live on disk rather than in the process, because the sweep exits between
runs.
Note for anyone editing `normalize-sweep.sh`: its file loop reads from a process substitution, not a pipe. A
pipe would put the loop in a subshell and every counter set inside it would be discarded on exit.
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
@@ -79,10 +126,70 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
"who downloaded what" audit trail for the files host.
## Alerting: Kibana evaluates, we deliver
A basic Elastic licence allows only two connector types, `.index` and `.server-log`. Everything that leaves
the stack — webhook, Slack, email, PagerDuty — is Gold or above and refuses with
`403 Action type .webhook is disabled because your basic license does not support it`. So Kibana cannot talk
to Discord, and the pipeline is split:
Kibana rule --.index connector--> vanguard-alerts --vanguard-alerts.timer--> Discord webhook
- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat
rule. Run it once, *after* Elasticsearch security is on.
- **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
them, and replayable after the fact.
- **`bin/vanguard-alerts.sh`** runs every minute, delivers anything newer than its cursor, and suppresses a
repeat of the same rule within 15 minutes as a safety net. No webhook configured means journal-only and a
non-zero exit, never a silent drop.
- **`bin/vanguard-deadman.sh`** watches both halves: a `kind=heartbeat` document arriving (Kibana still
evaluating and writing) and `alerts.state` being fresh (something still delivering). Either half dying
means no alarm can arrive, and nothing inside the stack could tell you.
### The rules themselves
`observability/kibana-alarms.py` creates them, so they are reviewable and reproducible rather than clicked in.
All of them repeat every **12 hours** while still true (`notify_when: onThrottleInterval`) and go silent when
the condition clears.
| Rule | Level | Fires when |
|---|---|---|
| Disk free below 15% | error | any real filesystem at ≥85% used. Pseudo filesystems are excluded by type, or every tmpfs would alarm forever |
| Memory free below 15% | warn | `system.memory.actual.used.pct` ≥ 0.85 |
| Discord bot not running | error | **no** metrics from the `discord-bot` container for 5 min (the one rule with a `<` comparator) |
| Discord bot unhealthy | error | the container healthcheck reports `unhealthy`, i.e. the bot's own `/health` says Discord is disconnected or the database does not answer |
| Recordings stuck waiting to be normalised | warn | `vanguard_media_pending_oldest_seconds` > 1 h |
| Normalise failures parked | warn | any file has a `normalize-failed.log` |
The two bot rules answer different questions and both are needed. "Not running" catches a stopped or
hard-crash-looping container. "Unhealthy" catches the worse case: the process is up, the port is open, and the
bot is doing nothing because its gateway or database is gone. That second one only works because the bot's
`/health` is exempt from its own identity middleware and returns 503 rather than 200 when degraded — a
healthcheck carries no Authentik headers, and a 200 with `discord_ready: false` would have made a dead bot
look healthy.
Elasticsearch, Kibana, Beat liveness and the disks are **not** here: that is the deadman's job, outside the
stack. Nothing that depends on Kibana can report that Kibana is dead, so duplicating those as rules would go
quiet in exactly the case that matters.
To add an alarm: author the rule in Kibana, point its action at the "Vanguard alert index" connector, and give
the document a `message` and a `level` of `warn` or `error`. Never set `kind: heartbeat` — that value is how
the delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
for real alarms; only the heartbeat should fire every interval.
## Retention
One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
Set by `configure.py --apply`, so nothing here needs clicking and nothing grows without bound:
| What | How long | Set by |
|---|---|---|
| `filebeat-*`, `metricbeat-*` | `LOG_RETENTION_DAYS`, default 30 | a **delete phase injected into the Beats' own ILM policies**. They ship a rollover but no delete, so indices otherwise grow until the disk watermark turns them read-only — which the deadman reports but cannot undo |
| `vanguard-alerts` | `ALERT_RETENTION_DAYS`, default 90 | `vanguard-alerts.sh`, a `_delete_by_query` at most once every 6 h. A heartbeat document arrives every minute, so this index needs pruning more than it needs ILM |
The delete phase is *injected* rather than written whole, so whatever rollover the Beats chose is preserved
across upgrades. Keeping `auth` and `deploy` lines longer than the rest would mean separate indices and a
second ILM policy; not worth it unless an audit asks.
## As built (Phase 13)
@@ -151,18 +258,55 @@ Discord notifier omitted, so nothing is half-configured and nothing fails to loa
- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
earlier entries are still on the host for `journalctl --boot=-1`.
- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it.
file and mount it, the way the gateway and dock pages are served, instead of inlining it. (The diag host no
longer has such a page at all: Kibana occupies its root.)
- **Filebeat 9 removed the `container` input.** It is not deprecated-but-working: the config is refused per
container with "Container input is deprecated. Use Filestream input with its container parser instead",
and Filebeat then runs happily with zero harvesters and creates no index at all. Container logs go through
`type: filestream` with `parsers: [container: ~]`, and every filestream input needs its own stable `id`.
This is exactly the failure the deadman's "up but not shipping" check exists to catch.
- **`add_docker_metadata` should be told where the container id is.** Its path-sniffing default did not
apply to our filestream paths, so the autodiscover template stamps `container.id` and the processor reads
it via `match_fields: ["container.id"]`. Without the id there are no `container.labels.*`, and without
those the `service.name` / `vanguard.category` promotion silently produces nothing.
- **A single node cannot place a replica**, so any beats index defaulting to one replica holds the cluster at
yellow permanently. Both beats set `setup.template.settings.index.number_of_replicas: 0` with
`setup.template.overwrite: true`; indices created before that need a one-time `_settings` call.
- **Kibana's Observability apps that check privileges cannot work with `xpack.security.enabled: false`.** The
Infrastructure / Metrics app calls `api/infra/host/count` on every view and Elasticsearch answers
`no handler found for uri [/_security/user/_has_privileges]`, surfaced as a red "error while fetching
resource" banner. The fix is to turn security ON, not to avoid the app. Doing so normally adds a password
prompt, which this stack forbids, so Kibana's **anonymous provider** signs every visitor in as one native
user instead (`observability/kibana.yml`). Security being on is also what any future read-only tier needs.
- **Moving a setting from a container environment variable into a config file can break it.** `SERVER_BASEPATH: ""`
was fine for years because the Kibana image drops empty env values, but `server.basePath: ""` in
`kibana.yml` is a validation error (`must start with a slash, don't end with one`) and Kibana refuses to
boot. When migrating env settings into a file, an empty value means "omit the key", not "set it to empty".
- **Metricbeat exits when Kibana is unreachable**, because `setup.dashboards.enabled: true` runs at startup.
A Metricbeat restart loop is often a symptom of a Kibana problem, not a Metricbeat one.
- **Alerting rules created while security was off hold no API key** and stop working once it is on, until each
rule is disabled and re-enabled. Create connectors and rules *after* security is on, not before.
- **Kibana's Observability charts group by `event.dataset`.** Beats do not set it, so without a processor
every line lands in one "unknown" bucket and the Logs rate chart is useless. Filebeat now fills it from the
container's `vanguard.service` label, or from `systemd.unit` for journald lines. `copy_fields` never
overwrites an existing target, so listing several sources for one field means first match wins.
- **Metricbeat field shapes differ per module on purpose.** The MediaMTX and Caddy scrapes are untyped, so a
value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
`use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
`.counter` and `.rate`. Look for the right one before concluding a metric is missing.
- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
## Wiring sketches (the options that were weighed)
## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
- **VictoriaLogs**: single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
(`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
behind the Caddy gate. Weakest at log search.
- **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a
single host's logs.
- **ELK** (chosen, and what is deployed): Elasticsearch with a pinned 2 GB heap, Kibana at the root of
`diag.BASE`, Filebeat and Metricbeat shipping into the same cluster so one dashboard can carry logs and
metrics together. The cost is the JVM heap and the index disk on a box that also runs x264; it was taken
because every lighter option needed three or four more pieces to reach a usable dashboard.

16
STATUS.md
View File

@@ -1,6 +1,6 @@
# Vanguard stack — what runs, what people do, what's unfinished
Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01.
Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).
@@ -59,8 +59,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **rebuilt 2026-11-03, deploy pending**: the first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
@@ -91,6 +94,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
(the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
from the bot itself now; an Authentik username is matched to one Discord member by name.
9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
@@ -111,3 +116,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
- Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.

156
bin/dumpd.py
View File

@@ -12,12 +12,17 @@ POST /me/clips -> {"device_id": "..."}: create clips/<user> (+ .stignore), add
POST /deploy -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from
/etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1;
otherwise logs the push and answers 202 "auto-deploy disabled".
POST /heartbeat -> Alertmanager's Watchdog route: touches a file the deadman timer checks, so a dead
alerting pipeline becomes visible (bin/vanguard-deadman.sh).
POST /heartbeat -> LEGACY, nothing calls this now. It was the Watchdog route when Alertmanager, and then a
Kibana webhook connector, posted here. A basic Elastic licence has no webhook connector, so
Kibana's rules write into the vanguard-alerts index instead and the deadman reads the
heartbeat from there. Kept because it costs nothing and is a ready-made target if an
external monitor is ever added.
POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle,
HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
GET /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret.
@@ -420,6 +425,140 @@ def verify_token(token):
return None
return user.lower()
# ---- GET /metrics -----------------------------------------------------------------------------------
# Prometheus text for Metricbeat's prometheus module. Two sources are merged:
# 1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
# 2. gauges computed here at scrape time: unpublished recordings, encodes in flight, failures, MediaMTX
# Only the loopback listener serves it, and Caddy routes no host to /metrics (every dumpd route over there is
# an explicit path matcher), so this is reachable from the host and from nowhere else. Nothing here is
# authenticated and nothing here is allowed to fail: a scrape must never be able to disturb the stack.
METRICS_DIR = Path(os.environ.get("VG_METRICS_DIR", str(MARK / "metrics")))
MTX_API = os.environ.get("MTX_API_URL", "http://127.0.0.1:9997")
IDLE_S = int(os.environ.get("MINS_IDLE", "3")) * 60 # the sweep's own "finished being written" rule
# counters key head -> (metric, type, help, the labels the remaining key fields fill, in order)
COUNTER_MAP = {
"jobs": ("vanguard_media_jobs_total", "counter", "normalise/deband jobs that finished", ("verdict", "outcome", "kind")),
"seconds": ("vanguard_media_job_seconds_total", "counter", "wall-clock seconds spent in ffmpeg", ("verdict", "kind")),
"runs": ("vanguard_media_runs_total", "counter", "times the script has started work", ()),
"deband": ("vanguard_media_deband_total", "counter", "timecode bands cropped off", ()),
"last_finish": ("vanguard_media_last_finish_timestamp_seconds", "gauge", "when the script last finished", ()),
"last_duration": ("vanguard_media_last_run_seconds", "gauge", "how long the last finished run took", ()),
}
def _num(value):
"""Prometheus wants a plain number: %g would turn a unix timestamp into 1.76e+09."""
return "%d" % value if float(value).is_integer() else "%.3f" % value
def _glob(*patterns):
found = []
for pattern in patterns:
try:
found.extend(ROOT.glob(pattern))
except OSError:
pass
return found
def mtx_paths():
"""MediaMTX's own view of what is live. None if it did not answer."""
try:
with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=1.5) as response:
return json.load(response).get("items") or []
except Exception:
return None
def metrics_text():
order, grouped = [], {}
def add(metric, typ, helptext, labels, value):
if metric not in grouped:
grouped[metric] = (typ, helptext, [])
order.append(metric)
grouped[metric][2].append((labels, value))
# 1. whatever the scripts have counted
for counters in sorted(METRICS_DIR.glob("*.counters")):
component = counters.name[: -len(".counters")]
try:
lines = counters.read_text().splitlines()
except OSError:
continue
for line in lines:
field = line.split()
if len(field) != 2:
continue
try:
value = float(field[1])
except ValueError:
continue
head, *rest = field[0].split(":")
if head not in COUNTER_MAP:
continue
metric, typ, helptext, names = COUNTER_MAP[head]
labels = {"component": component}
labels.update(dict(zip(names, rest)))
add(metric, typ, helptext, labels, value)
# 2. is each script executing right now? (its pid file exists only while it runs; a crash leaves a stale
# one, so the pid is checked against /proc rather than trusted)
for pid_file in sorted(METRICS_DIR.glob("*.pid")):
try:
pid = int(pid_file.read_text().strip() or 0)
except (OSError, ValueError):
pid = 0
add("vanguard_media_running", "gauge", "1 while the script is executing",
{"component": pid_file.name[: -len(".pid")]}, 1 if pid and Path(f"/proc/{pid}").is_dir() else 0)
# 3. the queue. Depth-limited globs on purpose: never a walk of the whole media tree.
now = _time.time()
recording, pending, oldest = 0, 0, 0.0
for f in _glob("streams/*/*/.live/*"):
if f.name.startswith(".") or not f.is_file():
continue
try:
age = now - f.stat().st_mtime
except OSError:
continue
# the sweep ignores anything touched in the last MINS_IDLE, so younger files are still being written
if age <= IDLE_S:
recording += 1
else:
pending += 1
oldest = max(oldest, age)
add("vanguard_media_recording", "gauge", "recordings MediaMTX is still writing into .live/", {}, recording)
add("vanguard_media_pending", "gauge", "finished recordings waiting to be normalised", {"kind": "stream"}, pending)
add("vanguard_media_pending_oldest_seconds", "gauge", "age of the oldest waiting recording", {}, round(oldest))
add("vanguard_media_encoding_in_flight", "gauge", "ffmpeg output files open right now", {},
len(_glob("streams/*/*.part", "streams/*/*/*.part", "streams/*/*/.live/*.part",
"clips/*/*.part", "clips/*/*/*.part", "clips/*/*/*/*.part")))
add("vanguard_media_failed_files", "gauge", "files parked with a normalize-failed log", {},
len(_glob("streams/*/*.normalize-failed.log", "streams/*/*/*.normalize-failed.log",
"clips/*/*.normalize-failed.log", "clips/*/*/*.normalize-failed.log",
"clips/*/*/*/*.normalize-failed.log")))
# 4. MediaMTX. The Prometheus scrape of :9998 already has per-path detail; this is the count the portal
# and the dock actually mean by "live", with the derived .webrtc audio mirrors separated out.
paths = mtx_paths()
add("vanguard_mediamtx_up", "gauge", "did the MediaMTX API answer this scrape", {}, 0 if paths is None else 1)
if paths is not None:
ready = [p for p in paths if p.get("ready")]
add("vanguard_streams_active", "gauge", "streams publishing right now, mirrors excluded", {},
sum(1 for p in ready if not str(p.get("name", "")).endswith(".webrtc")))
add("vanguard_streams_mirrors", "gauge", "derived .webrtc audio mirrors running", {},
sum(1 for p in ready if str(p.get("name", "")).endswith(".webrtc")))
add("vanguard_stream_readers", "gauge", "viewers attached across all paths", {},
sum(len(p.get("readers") or []) for p in ready))
out = []
for metric in order:
typ, helptext, series = grouped[metric]
if helptext:
out.append(f"# HELP {metric} {helptext}")
out.append(f"# TYPE {metric} {typ}")
for labels, value in series:
tags = ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
out.append(f"{metric}{{{tags}}} {_num(value)}" if tags else f"{metric} {_num(value)}")
return "\n".join(out) + "\n"
class H(BaseHTTPRequestHandler):
server_version = "dumpd/1"
def _json(self, code, obj):
@@ -429,6 +568,13 @@ class H(BaseHTTPRequestHandler):
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _text(self, code, body):
raw = body.encode()
self.send_response(code)
self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def _deny(self):
self._json(401, {"error": "not authenticated (no identity header from the proxy)"})
def _parse(self):
@@ -457,6 +603,12 @@ class H(BaseHTTPRequestHandler):
def do_GET(self):
if self.server.server_port == AUTH_LISTEN[1]:
return self._json(404, {"error": "not found"}) # the auth port serves POST /mtx-auth only
if self.path == "/metrics":
try:
return self._text(200, metrics_text())
except Exception as error: # a scrape must never take dumpd down
log("metrics_failed", level="error", error=str(error))
return self._text(500, f"# metrics unavailable: {error}\n")
if self.path.startswith("/time"):
# server clock for the timecode strip (the /clock/ page and Go Live measure their offset against this)
return self._json(200, {"now_ms": int(_time.time() * 1000)})

35
bin/normalize-sweep.sh
View File

@@ -36,6 +36,15 @@ MAX_W=2560; MAX_H=1440; MAX_FPS=60
exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
flock -n 9 || exit 0
# Metrics (LOGGING.md): counters for dumpd's /metrics. Optional on purpose — if the helper is not installed
# yet, these become no-ops and normalising carries on exactly as before.
if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
VG_MET_COMPONENT=sweep
. "$(dirname "$0")/vgmetrics.sh" && vgmet_init
else
vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
fi
# logfmt (LOGGING.md): vglog <level> event=<verb> key=value ... values with spaces must arrive pre-quoted
vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=sweep %s\n' "$(date -Is)" "$lvl" "$*"; }
log() { vglog info "msg=\"$*\""; }
@@ -84,9 +93,6 @@ probe() {
echo "ok"
}
find "$ROOT/streams" "$ROOT/clips" -type f \
\( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null |
while IFS= read -r -d '' src; do
base="${src%.*}"; ext="${src##*.}"
live=0; stream=""
@@ -121,6 +127,7 @@ while IFS= read -r -d '' src; do
[[ -e "$out" ]] && continue
fi
tmp="$out.part"
kind=clip; (( live )) && kind=stream # metric label: stream encodes are the expensive ones
vglog info "event=verdict verdict=$verdict file=\"$src\" reason=\"$reason\" stream=\"$stream\""
tcargs=(); tcjson=""; band=0; deband=0
if (( live )); then
@@ -157,9 +164,11 @@ PYEOF
fi
fi
fi
job_t0=$(vgmet_time)
err=$(mktemp /tmp/vgsweep-err.XXXXXX) # ffmpeg's own words; rc= alone is not diagnosable
if [[ $verdict == remux ]]; then
nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
-map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
-map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
else
# scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
@@ -175,11 +184,14 @@ PYEOF
-c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \
-vf "$vf" -fps_mode "$fpsmode" "${fpsargs[@]}" \
"${aargs[@]}" \
"${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
"${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
fi
rc=$?
job_secs=$(vgmet_since "$job_t0")
d1=$(dur "$src"); d2=$(dur "$tmp")
if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
vgmet_add "jobs:$verdict:done:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
(( deband )) && vgmet_add deband 1
if (( live )); then
mv -f "$tmp" "$out"; chown 1000:1000 "$out" 2>/dev/null
if [[ -n $tcjson ]]; then # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
@@ -192,6 +204,7 @@ PYEOF
rm -f "$src" # band removal: the cropped file IS the recording; raw-with-band is not kept
fi
rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true # drop .live and the empty title folder
rm -f "$err"
vglog info "event=published file=\"$out\" stream=\"$stream\" verdict=$verdict"; continue
elif (( synced )); then
rel_src="${src#"$stroot"/}"; rel_out="${out#"$stroot"/}"
@@ -211,7 +224,13 @@ PYEOF
vglog info "event=done file=\"$out\" verdict=$verdict"
else
rm -f "$tmp"
{ echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"; } > "$base.normalize-failed.log"
vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc log=\"$base.normalize-failed.log\"" >&2
vgmet_add "jobs:$verdict:failed:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
{ echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"
echo "--- ffmpeg stderr ---"; tail -40 "$err" 2>/dev/null; } > "$base.normalize-failed.log"
why=$(tail -1 "$err" 2>/dev/null | tr -d '"' | cut -c1-200)
vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc reason=\"${why:-no stderr}\" log=\"$base.normalize-failed.log\"" >&2
fi
done
rm -f "$err"
done < <(find "$ROOT/streams" "$ROOT/clips" -type f \
\( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null)

152
bin/vanguard-alerts.sh Executable file
View File

@@ -0,0 +1,152 @@
#!/bin/bash
# Deliver Kibana's alerts to Discord. Runs from a systemd timer every minute; curl and python only.
#
# WHY THIS EXISTS
# On a basic license Kibana's only connector types are .index and .server-log — every connector that leaves
# the stack is Gold or above. So Kibana rules write their alerts into ONE Elasticsearch index (the alert bus,
# created by observability/kibana-setup.py) and this script is the delivery side. Alerts are therefore
# ordinary documents: queryable, replayable, and visible in Discover alongside the logs that caused them.
#
# Heartbeat documents (kind=heartbeat) are deliberately NOT delivered. They exist for bin/vanguard-deadman.sh
# to notice their absence, which is the only way to catch the alerting pipeline itself dying.
#
# Repeat suppression is a safety net, not the primary throttle: a rule should use "On status change" or its
# own throttle in Kibana. This stops a misconfigured rule turning into a message every minute, and it is per
# rule name, so one noisy rule never silences another.
#
# Exit status: 0 when there was nothing to send or everything was sent; 1 when alerts existed and could not be
# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
set -uo pipefail
ES="${ES_URL:-http://127.0.0.1:9200}"
ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
INDEX="${ALERT_INDEX:-vanguard-alerts}"
STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"
WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
SUPPRESS_S="${ALERT_SUPPRESS_S:-900}"
BATCH="${ALERT_BATCH:-50}"
vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=alerts %s\n' "$(date -Is)" "$lvl" "$*"; }
es_curl() { if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi; }
state_get() { [[ -r $STATE ]] || return 0; grep -m1 "^$1=" "$STATE" 2>/dev/null | cut -d= -f2-; }
state_set() {
local key="$1" value="$2" tmp
install -d "$(dirname "$STATE")" 2>/dev/null || return 0
tmp=$(mktemp "${STATE}.XXXXXX" 2>/dev/null) || return 0
{ [[ -r $STATE ]] && grep -v "^${key}=" "$STATE"; printf '%s=%s\n' "$key" "$value"; } > "$tmp" 2>/dev/null
mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
}
webhook=""
[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE")
# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like
# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit.
case "${webhook:-}" in
"") ;;
https://*) ;;
*) vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;;
esac
now=$(date +%s)
cursor=$(state_get cursor); cursor="${cursor:-0}"
# Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely.
query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor")
response=$(es_curl -m 15 -H 'Content-Type: application/json' "$ES/$INDEX/_search" -d "$query")
if [[ -z $response ]]; then
vglog error 'event=unreachable msg="Elasticsearch did not answer; cannot read the alert bus"' >&2
exit 1
fi
# A missing index just means no rule has fired yet. Not a fault, and not worth a line every minute.
if grep -q 'index_not_found_exception' <<<"$response"; then
exit 0
fi
# Retention. A heartbeat document arrives every minute, so without this the alert bus grows without bound.
# At most once every 6 h: a delete_by_query is cheap, but not worth doing sixty times an hour. Deliberately
# before the "nothing new" exit below, because most runs have nothing to deliver and would skip it.
retention_days="${ALERT_RETENTION_DAYS:-90}"
last_purge=$(state_get purged); last_purge="${last_purge:-0}"
if (( now - last_purge > 21600 )); then
purge=$(es_curl -m 120 -H 'Content-Type: application/json' \
-XPOST "$ES/$INDEX/_delete_by_query?conflicts=proceed" \
-d "{\"query\":{\"range\":{\"@timestamp\":{\"lt\":\"now-${retention_days}d\"}}}}")
gone=$(printf '%s' "${purge:-}" | python3 -c 'import json,sys
try: print(json.load(sys.stdin).get("deleted", 0))
except Exception: print("?")')
state_set purged "$now"
vglog info "event=purged older_than=${retention_days}d deleted=${gone:-?}"
fi
rows=$(printf '%s' "$response" | python3 -c '
import json, sys
from datetime import datetime
try:
data = json.load(sys.stdin)
except Exception:
sys.exit(0)
for hit in data.get("hits", {}).get("hits", []):
source = hit.get("_source", {})
stamp = str(source.get("@timestamp", ""))
try:
when = datetime.fromisoformat(stamp.replace("Z", "+00:00"))
millis = int(when.timestamp() * 1000)
except Exception:
continue
def clean(value, limit):
return " ".join(str(value).split())[:limit] or "-"
print("\t".join((str(millis),
clean(source.get("rule", "unnamed rule"), 120),
clean(source.get("level", "warn"), 12),
clean(source.get("message", json.dumps(source)[:400]), 1200))))
')
[[ -z $rows ]] && exit 0
# One per line: "a=0 b=0 now" would make bash run `now` as a COMMAND with a and b as temporary
# assignments, leaving every counter unset for the rest of the script.
sent=0
suppressed=0
failed=0
while IFS=$'\t' read -r millis rule level message; do
[[ -z ${millis:-} ]] && continue
last=$(state_get "sent:$rule"); last="${last:-0}"
if (( now - last < SUPPRESS_S )); then
suppressed=$((suppressed + 1))
vglog info "event=suppressed rule=\"$rule\" within=${SUPPRESS_S}s"
state_set cursor "$millis"
continue
fi
case "$level" in
error|critical) mark="[ERROR]" ;;
warn|warning) mark="[WARN]" ;;
*) mark="[$level]" ;;
esac
text="$mark $rule — $message"
if [[ -z $webhook ]]; then
vglog warn "event=undeliverable rule=\"$rule\" msg=\"no Discord webhook configured; alert only in the journal\"" >&2
vglog warn "event=alert rule=\"$rule\" level=$level msg=\"$message\"" >&2
failed=$((failed + 1))
state_set cursor "$millis"
continue
fi
payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))')
# curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000.
code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \
-d "$payload" "$webhook" || true)
code="${code:-000}"
if [[ $code == 2* ]]; then
sent=$((sent + 1)); state_set "sent:$rule" "$now"
vglog info "event=delivered rule=\"$rule\" level=$level"
else
failed=$((failed + 1))
vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2
fi
state_set cursor "$millis"
done <<<"$rows"
vglog info "event=run sent=$sent suppressed=$suppressed failed=$failed"
(( failed > 0 )) && exit 1
exit 0

47
bin/vanguard-deadman.sh
View File

@@ -25,9 +25,19 @@
set -uo pipefail
ES="${ES_URL:-http://127.0.0.1:9200}"
# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
# is missing, the calls below get 401 and the deadman reports Elasticsearch as unreachable — which is correct,
# because from its point of view it is. Kibana needs nothing: its anonymous provider signs callers in.
ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
es_curl() {
if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi
}
KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
HEARTBEAT="${ALERT_HEARTBEAT_FILE:-/srv/media/.markers/alert-heartbeat}"
ALERT_INDEX="${ALERT_INDEX:-vanguard-alerts}" # the alert bus Kibana's rules write into
ALERT_STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}" # touched by every vanguard-alerts.sh run
ALERT_STATE_MAX_S="${ALERT_STATE_MAX_S:-900}"
HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
@@ -44,7 +54,7 @@ for k in sys.argv[1].split("."):
print(d)' "$1" 2>/dev/null; }
# 1. Elasticsearch
health=$(curl -s -m 8 "$ES/_cluster/health" || true)
health=$(es_curl -m 8 "$ES/_cluster/health" || true)
if [[ -z $health ]]; then
note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
else
@@ -56,7 +66,7 @@ else
esac
# 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
blocked=$(curl -s -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
blocked=$(es_curl -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
| python3 -c 'import json,sys
try: d=json.load(sys.stdin)
except Exception: sys.exit(0)
@@ -66,7 +76,7 @@ print(len(bad))' 2>/dev/null || echo 0)
# 4 and 5. are the Beats actually writing?
for beat in filebeat metricbeat; do
count=$(curl -s -m 8 -H 'Content-Type: application/json' \
count=$(es_curl -m 8 -H 'Content-Type: application/json' \
"$ES/${beat}-*/_count" \
-d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
if [[ -z ${count:-} ]]; then
@@ -81,12 +91,31 @@ fi
level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
[[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
# 6. the heartbeat a Kibana rule posts here. Missing right after a restart is not yet a fault; age is.
if [[ -f $HEARTBEAT ]]; then
age=$(( $(date +%s) - $(stat -c %Y "$HEARTBEAT") ))
(( age <= HEARTBEAT_MAX_S )) || note "no alert heartbeat for ${age}s (the Kibana rule → webhook → dumpd path is broken)"
# 6. the heartbeat. A Kibana rule writes a kind=heartbeat document into the alert index every minute (see
# observability/kibana-setup.py). Its absence is the ONLY thing that catches the alerting pipeline itself
# dying: if Kibana, its rule engine or the index connector stops, no ordinary alert can report it, because
# the thing that would report it is what broke. Checked in Elasticsearch, not as a file: a basic licence has
# no webhook connector, so nothing posts to dumpd any more.
if [[ -n $health ]]; then
beats=$(es_curl -m 8 -H 'Content-Type: application/json' "$ES/$ALERT_INDEX/_count" \
-d "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"kind\":\"heartbeat\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-${HEARTBEAT_MAX_S}s\"}}}]}}}")
if grep -q 'index_not_found_exception' <<<"${beats:-}"; then
note "the alert index $ALERT_INDEX does not exist — the Kibana connector and heartbeat rule have not been created (observability/kibana-setup.py)"
else
note "no alert heartbeat has ever arrived (is the Kibana heartbeat rule created? README 5h)"
hb=$(printf '%s' "${beats:-}" | jq_get count)
if [[ -z ${hb:-} ]]; then
note "cannot read the alert heartbeat (query against $ALERT_INDEX failed)"
elif (( hb == 0 )); then
note "no alert heartbeat in the last $((HEARTBEAT_MAX_S/60)) minutes — Kibana's rule engine or the index connector has stopped, so NO alert can reach Discord"
fi
fi
fi
# 7. is the delivery side running? The heartbeat proves Kibana still writes alerts; this proves something
# still takes them to Discord. Both halves have to be alive for an alarm to arrive.
if [[ -f $ALERT_STATE ]]; then
age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
(( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
fi
# the disks, independently of any agent

14
bin/vgdeband.sh
View File

@@ -11,6 +11,13 @@ ROOT="${MEDIA_ROOT:-/srv/media}"
vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgdeband %s\n' "$(date -Is)" "$lvl" "$*"; }
log() { vglog info "msg=\"$*\""; }
dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
# Metrics (LOGGING.md): the same counters the sweep keeps, under component=vgdeband. No-ops if not installed.
if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
VG_MET_COMPONENT=vgdeband
. "$(dirname "$0")/vgmetrics.sh" && vgmet_init
else
vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
fi
if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi
for f in "${files[@]}"; do
side="${f%.*}.json"
@@ -26,19 +33,22 @@ print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode
tmp="$f.part"
tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime")
vglog info "event=deband file=\"$f\" width=$w band_rows=$band timecode=\"$tc\""
job_t0=$(vgmet_time)
nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \
-map 0:v:0 -map '0:a?' \
-c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "${ENCODE_THREADS:-2}" \
-vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \
"${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
rc=$?; d1=$(dur "$f"); d2=$(dur "$tmp")
rc=$?; job_secs=$(vgmet_since "$job_t0"); d1=$(dur "$f"); d2=$(dur "$tmp")
vgmet_add "seconds:encode:stream" "$job_secs"
if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
vgmet_add "jobs:encode:done:stream"; vgmet_add deband 1
mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null
python3 -c '
import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True
json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null
vglog info "event=done file=\"$f\""
else
rm -f "$tmp"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2
rm -f "$tmp"; vgmet_add "jobs:encode:failed:stream"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2
fi
done

63
bin/vgmetrics.sh Normal file
View File

@@ -0,0 +1,63 @@
# Shared metric counters for the media scripts. Sourced, never executed.
#
# VG_MET_COMPONENT=sweep # a plain assignment, NOT a prefix on the next line:
# . /usr/local/bin/vgmetrics.sh && vgmet_init # bash discards "VAR=x . file" assignments on return
# vgmet_add "jobs:encode:done:stream" # +1
# vgmet_add "seconds:encode:stream" 41.2 # +41.2
#
# Counters accumulate across runs in MARK/metrics/<component>.counters as plain "key value" lines, and a
# <component>.pid file exists only while the script is executing. dumpd turns both into Prometheus text on
# its /metrics; nothing here renders or serves anything, so the scripts stay simple and there is one place
# that knows the metric names (bin/dumpd.py COUNTER_MAP).
#
# Keys are "name:label:label..." — the name picks the metric, the rest fill its labels in order.
# Writes are atomic (temp file + mv) and every failure is silent: metrics must never break a sweep.
VG_MET_DIR="${VG_MET_DIR:-${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
VG_MET_COMPONENT="${VG_MET_COMPONENT:-$(basename "${0%.sh}")}"
VG_MET_START=0
declare -A VGMET=()
vgmet_init() {
: "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
: "${VG_MET_COMPONENT:=unknown}"
VG_MET_START=$(date +%s)
mkdir -p "$VG_MET_DIR" 2>/dev/null || return 0
local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" k v
if [[ -r $f ]]; then
while read -r k v; do [[ -n ${k:-} && -n ${v:-} ]] && VGMET["$k"]="$v"; done < "$f"
fi
vgmet_add runs 1
printf '%s\n' "$$" > "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
trap vgmet_flush EXIT
}
# vgmet_add <key> [amount] — float-safe accumulate
vgmet_add() {
local k="$1" v="${2:-1}"
VGMET["$k"]=$(awk -v a="${VGMET[$k]:-0}" -v b="$v" 'BEGIN{ printf "%.3f", a+b }')
}
vgmet_set() { VGMET["$1"]="$2"; }
vgmet_flush() {
: "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
: "${VG_MET_COMPONENT:=unknown}"
local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" tmp now k
rm -f "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
[[ -d $VG_MET_DIR ]] || return 0
now=$(date +%s)
vgmet_set last_finish "$now"
vgmet_set last_duration "$(( now - VG_MET_START ))"
tmp=$(mktemp "$VG_MET_DIR/.$VG_MET_COMPONENT.XXXXXX" 2>/dev/null) || return 0
for k in "${!VGMET[@]}"; do printf '%s %s\n' "$k" "${VGMET[$k]}"; done > "$tmp" 2>/dev/null
chmod 0644 "$tmp" 2>/dev/null
mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" 2>/dev/null
}
# start/elapsed for one job
vgmet_time() { date +%s.%N; }
vgmet_since() { awk -v s="$1" -v e="$(date +%s.%N)" 'BEGIN{ printf "%.3f", e-s }'; }
# No-op fallbacks are defined by the callers if this file is missing, so a half-finished deploy cannot
# break normalising. Keep every function above failure-tolerant.

22
check.sh
View File

@@ -9,11 +9,11 @@ for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh register
[[ -f $f ]] || continue
bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; }
done
for f in configure.py bin/dumpd.py bin/vgstrip.py; do
for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py observability/kibana-alarms.py; do
python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
done
rm -rf __pycache__ bin/__pycache__
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
node -e "
const fs=require('fs');
for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
@@ -27,6 +27,24 @@ c.write_mediamtx('media.example.com'); yaml.safe_load(open('mediamtx.yml')); c.w
assert 'authMethod: http' in open('mediamtx.yml').read(); assert 'idp-h-usr' in open('copyparty.conf').read()
" ) && say "configure.py generators" ok || { say "configure.py generators" "FAILED"; fail=1; }
rm -rf "$T"
# dumpd GET /metrics: valid Prometheus text out of script counters + a fake media tree
T=$(mktemp -d); mkdir -p "$T/.markers/metrics" "$T/streams/u/Title/.live"
printf 'runs 2\njobs:encode:done:stream 1\nseconds:encode:stream 12.5\nlast_finish 1762210000\n' > "$T/.markers/metrics/sweep.counters"
touch -d '2020-01-01' "$T/streams/u/Title/.live/old.mp4"
if MEDIA_ROOT="$T" MTX_API_URL=http://127.0.0.1:1 python3 - <<'PYMET' >/dev/null 2>&1
import re, sys
sys.path.insert(0, "bin")
import dumpd
text = dumpd.metrics_text()
for line in text.splitlines():
assert line.startswith("#") or re.fullmatch(r'[a-z_]+(\{[^}]*\})? -?[0-9.]+', line), line
assert 'vanguard_media_jobs_total{component="sweep",kind="stream",outcome="done",verdict="encode"} 1' in text
assert 'vanguard_media_last_finish_timestamp_seconds{component="sweep"} 1762210000' in text # not 1.76e+09
assert 'vanguard_media_pending{kind="stream"} 1' in text
assert 'vanguard_mediamtx_up 0' in text and 'vanguard_streams_active' not in text # no fake zeros
PYMET
then say "dumpd /metrics" ok; else say "dumpd /metrics" "FAILED"; fail=1; fi
rm -rf "$T" bin/__pycache__
if command -v caddy >/dev/null; then
T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('media.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com','f.example.com','s.example.com','c.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; }
rm -rf "$T"

207
configure.py
View File

@@ -24,7 +24,8 @@ Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/va
Discord bot token/guild), folders, restarts
Set VG_DRY=1 to write files only (no sudo / docker / folders).
"""
import getpass, hashlib, os, re, secrets, shutil, subprocess, sys
import base64, getpass, hashlib, json, os, re, secrets, shutil, subprocess, sys, time
import urllib.error, urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
@@ -390,8 +391,23 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
env.setdefault("BOT_DEPLOY_AUTO", "1") # 0 = refuse bundles posted by CI; install by hand instead
env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID") # the only .env keys the bot's CI may set
env.setdefault("DIAG_REQUIRE_GROUP", "media-admins") # who may see logs, metrics and alerts
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook for Kibana rules AND the deadman
env.setdefault("ALERT_WEBHOOK_URL", "") # Discord webhook: vanguard-alerts.sh delivery AND the deadman
env.setdefault("KIBANA_ENCRYPTION_KEY", secrets.token_hex(32)) # without a fixed key Kibana loses saved objects on restart
# Elasticsearch security is ON (observability/kibana.yml explains why, and why nobody sees a login).
# These are service credentials, never typed by a person: Discord remains the only human login.
def pw(): return "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32]
env.setdefault("ES_ELASTIC_PASSWORD", pw()) # the elastic superuser; bootstrapped on first secured start
env.setdefault("ES_KIBANA_PASSWORD", pw()) # kibana_system: Kibana's own internal account
env.setdefault("VG_BEATS_USER", "vanguard_beats")
env.setdefault("VG_BEATS_PASSWORD", pw()) # what Filebeat and Metricbeat ship as
env.setdefault("VG_DIAG_USER", "vanguard_diag") # who every Kibana visitor silently becomes
env.setdefault("VG_DIAG_PASSWORD", pw())
# What that visitor may do. superuser keeps today's behaviour exactly (everyone past the gate is an admin).
# Narrowing this to "editor" is step one of the read-only tier in STATUS.md 9b.
env.setdefault("DIAG_KIBANA_ROLES", "superuser")
env.setdefault("ALERT_INDEX", "vanguard-alerts") # the alert bus: Kibana writes, vanguard-alerts.sh delivers
env.setdefault("LOG_RETENTION_DAYS", "30") # ILM delete phase for filebeat-*/metricbeat-*
env.setdefault("ALERT_RETENTION_DAYS", "90") # how long delivered alerts stay queryable
env.setdefault("DISCORD_BOT_TOKEN", "") # fill in: Discord dev portal -> the Authentik app -> Bot
env.setdefault("DISCORD_GUILD_ID", "")
env["WATCH_HOST"] = watch_host or "watch.example.com"
@@ -421,7 +437,7 @@ rtspAddress: 127.0.0.1:8554
rtspTransports: [tcp]
rtspEncryption: "no"
moq: no
metrics: yes # Prometheus text format for whichever metrics store is chosen (LOGGING.md); loopback only
metrics: yes # Prometheus text format, scraped by Metricbeat (LOGGING.md); loopback only
metricsAddress: :9998
pprof: no
@@ -559,9 +575,10 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
e = read_env()
Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
# 0640 root:65534 — the alertmanager image runs as nobody(65534) and the deadman runs as root. A webhook
# URL is a credential for that channel, so it is not world-readable.
run(["sudo", "install", "-m", "0640", "-o", "root", "-g", "65534", "alert-webhook", "/etc/vanguard/alert-webhook"])
# 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be
# group-readable by nobody(65534) for the Alertmanager container, which no longer exists.
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"]) # read by dumpd (/deploy webhook) and deploy.sh
if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
@@ -577,13 +594,27 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
Path("99-vanguard-elasticsearch.conf").write_text("vm.max_map_count=262144\n")
run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state
run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"]) # script counters -> dumpd /metrics
# The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
# /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
"bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
if media:
run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
if Path("bin/vgmetrics.sh").exists():
run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband
if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
# The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
run(["sudo", "systemctl", "restart", "dumpd"])
print("== containers")
dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"]
dc = compose_cmd()
# --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how
# the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them.
run(dc + ["up", "-d", "--build", "--remove-orphans"]) # recreates only services whose definition changed
@@ -594,10 +625,138 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
print(" run docker compose restart mediamtx between streams (cuts a live recording). Adding/removing")
print(" streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
def compose_cmd():
"""docker compose, with sudo only if this user cannot reach the daemon."""
if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:
return ["docker", "compose"]
return ["sudo", "docker", "compose"]
def es_security(dc):
"""Create Elasticsearch's native users after the cluster is up. Idempotent; safe to re-run.
Security is on, so three accounts have to exist before Kibana or the Beats can work:
kibana_system Kibana's internal account (built in; only its password needs setting)
VG_BEATS_USER what Filebeat and Metricbeat authenticate as
VG_DIAG_USER who every Kibana visitor silently becomes (kibana.yml's anonymous provider)
The elastic superuser itself is bootstrapped by the container from ELASTIC_PASSWORD, which only works
while the cluster has never had security on. If that did not take, this says exactly what to run rather
than guessing: a reset would desync the password the compose healthcheck uses.
"""
env = read_env()
elastic_pw = env.get("ES_ELASTIC_PASSWORD", "")
if not elastic_pw:
print(" elasticsearch: no ES_ELASTIC_PASSWORD in .env (write_env not run?) — skipped"); return
if DRY:
print(" (dry) elasticsearch: set kibana_system + create "
f"{env.get('VG_BEATS_USER')} and {env.get('VG_DIAG_USER')}"); return
base = "http://127.0.0.1:9200"
def call(method, path, body=None):
request = urllib.request.Request(base + path, method=method,
data=None if body is None else json.dumps(body).encode())
request.add_header("Content-Type", "application/json")
request.add_header("Authorization", "Basic " + base64.b64encode(f"elastic:{elastic_pw}".encode()).decode())
with urllib.request.urlopen(request, timeout=20) as response:
raw = response.read()
return json.loads(raw) if raw else {}
status = None
for _ in range(45): # the cluster can take a while on a box busy with x264
try:
call("GET", "/_cluster/health"); status = "ok"; break
except urllib.error.HTTPError as error:
status = error.code
if error.code == 401: break # answering, but our password is not the one it bootstrapped
time.sleep(4)
except Exception:
time.sleep(4)
if status == 401:
print(" elasticsearch: REFUSED the password in .env. The cluster already had security enabled, so")
print(" ELASTIC_PASSWORD no longer bootstraps anything. Set it to the real one:")
print(" docker compose exec elasticsearch bin/elasticsearch-reset-password -u elastic -b -s")
print(" put that value in .env as ES_ELASTIC_PASSWORD, then re-run --apply.")
return
if status != "ok":
print(" elasticsearch: not answering yet — re-run ./configure.py --apply once it is up"); return
try:
# If security is somehow still off, every call below 400s with "no handler found for uri". Check once
# and say so plainly, rather than reporting a confusing failure on the first password write.
call("POST", "/_security/user/_has_privileges", {"cluster": ["monitor"]})
except urllib.error.HTTPError as error:
if error.code == 400:
print(" elasticsearch: the security API is absent — this cluster is still running with")
print(" xpack.security.enabled=false. Recreate it: docker compose up -d elasticsearch")
return
try:
call("POST", "/_security/user/kibana_system/_password", {"password": env["ES_KIBANA_PASSWORD"]})
print(" elasticsearch: kibana_system password set")
for user, password, roles in (
(env["VG_BEATS_USER"], env["VG_BEATS_PASSWORD"], ["superuser"]),
(env["VG_DIAG_USER"], env["VG_DIAG_PASSWORD"],
[r.strip() for r in env.get("DIAG_KIBANA_ROLES", "superuser").split(",") if r.strip()]),
):
call("PUT", f"/_security/user/{user}", {"password": password, "roles": roles,
"full_name": "Vanguard service account"})
print(f" elasticsearch: user {user} -> {', '.join(roles)}")
except urllib.error.HTTPError as error:
print(f" elasticsearch: user setup failed ({error.code}) — {error.read()[:200].decode(errors='replace')}")
return
# The alert bus. Kibana's rules write here through the .index connector (the only outward connector a
# basic licence allows), bin/vanguard-alerts.sh delivers from here to Discord, and the deadman watches
# for the heartbeat document. Pre-created with an explicit mapping so "level" and "rule" are keywords
# and therefore aggregatable, rather than whatever dynamic mapping the first document implies.
alert_index = env.get("ALERT_INDEX", "vanguard-alerts")
try:
call("PUT", f"/{alert_index}", {
"settings": {"number_of_replicas": 0},
"mappings": {"properties": {
"@timestamp": {"type": "date"},
"kind": {"type": "keyword"},
"rule": {"type": "keyword"},
"level": {"type": "keyword"},
"service": {"type": "keyword"},
"message": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 1024}}},
}}})
print(f" elasticsearch: alert index {alert_index} created")
except urllib.error.HTTPError as error:
body = error.read().decode(errors="replace")
print(f" elasticsearch: alert index {alert_index} already there"
if "resource_already_exists" in body else f" elasticsearch: alert index failed — {body[:160]}")
# Retention. The Beats create their own ILM policies with a rollover but NO delete phase, so indices
# grow until the disk watermark turns them read-only — which the deadman reports but cannot undo. Inject
# a delete phase instead of writing a whole policy, so whatever the Beats chose for rollover is kept.
keep_days = env.get("LOG_RETENTION_DAYS", "30")
for policy in ("filebeat", "metricbeat"):
try:
current = call("GET", f"/_ilm/policy/{policy}")[policy]["policy"]
except (urllib.error.HTTPError, KeyError):
print(f" elasticsearch: ILM policy {policy} not created yet — re-run --apply after the beat has started")
continue
phases = current.setdefault("phases", {})
if phases.get("delete", {}).get("min_age") == f"{keep_days}d":
print(f" elasticsearch: {policy} already deletes after {keep_days}d")
continue
phases["delete"] = {"min_age": f"{keep_days}d", "actions": {"delete": {}}}
call("PUT", f"/_ilm/policy/{policy}", {"policy": current})
print(f" elasticsearch: {policy} indices now deleted after {keep_days}d")
# The deadman runs on the host as root and must authenticate too. netrc, not -u, so the password never
# shows up in anyone's process list.
Path("es-netrc").write_text(f"machine 127.0.0.1 login elastic password {elastic_pw}\n")
run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "es-netrc", "/etc/vanguard/es-netrc"])
Path("es-netrc").unlink(missing_ok=True)
# Kibana and the Beats were started before those accounts existed, so they are sitting in a retry loop.
run(dc + ["restart", "kibana", "filebeat", "metricbeat"])
print(" elasticsearch: netrc written for the deadman; kibana + beats restarted")
def post_start_fixups(dc):
"""Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin."""
env = read_env()
print("== post-start fixups (idempotent)")
es_security(dc) # before anything else: Kibana and the Beats cannot authenticate until this runs
# Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it
cfg = Path("/srv/media/.syncthing/config.xml")
if not DRY and cfg.exists():
@@ -627,6 +786,32 @@ def post_start_fixups(dc):
print(f" gitea: admin user {env['GITEA_ADMIN_USER']} created (password in .env)")
else: print(" gitea: ok" if r.returncode == 0 else " gitea: not ready (install page or db?) — run ./configure.py --apply again later")
else: print(" (dry) gitea admin user")
# Kibana first-run state: the alert connector, the heartbeat rule and the data view. Without this a
# fresh install has no way to deliver an alert and the deadman alarms about its own missing heartbeat.
# It needs an API key because Basic is not an accepted scheme for this Kibana (anonymous is the only
# enabled provider), and it waits for Kibana itself, which es_security restarted a moment ago.
setup = Path("observability/kibana-setup.py")
if setup.exists() and env.get("ES_ELASTIC_PASSWORD"):
if DRY:
print(" (dry) kibana: connector, heartbeat rule, data view")
else:
try:
request = urllib.request.Request(
"http://127.0.0.1:9200/_security/api_key", method="POST",
data=json.dumps({"name": "vanguard-setup", "expiration": "10m"}).encode())
request.add_header("Content-Type", "application/json")
request.add_header("Authorization", "Basic " + base64.b64encode(
f"elastic:{env['ES_ELASTIC_PASSWORD']}".encode()).decode())
with urllib.request.urlopen(request, timeout=20) as response:
api_key = json.load(response)["encoded"]
result = subprocess.run([sys.executable, str(setup)], text=True, capture_output=True,
env={**os.environ, "KEY": api_key,
"ALERT_INDEX": env.get("ALERT_INDEX", "vanguard-alerts")})
for line in (result.stdout + result.stderr).strip().splitlines():
print(" kibana:", line)
except Exception as error:
print(f" kibana: first-run setup skipped ({error}) — run it by hand, see its docstring")
# Jellyfin plugin from the committed zip
if Path("installJellyfinPlugin.sh").exists():
if DRY: print(" (dry) installJellyfinPlugin.sh")
@@ -641,8 +826,9 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}")
print(f"== live : https://{live_host}/ (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live : (not set — run --full)")
alert_hook = read_env().get("ALERT_WEBHOOK_URL", "")
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); Kibana rules and the deadman post there" if alert_hook
else "NO WEBHOOK — Kibana has nowhere to send and the deadman only writes to the journal. Set ALERT_WEBHOOK_URL in .env."))
print("== alerts : " + (f"Discord webhook set ({alert_hook[:38]}…); vanguard-alerts.timer and the deadman post there" if alert_hook
else "NO WEBHOOK — Kibana still records alerts in the vanguard-alerts index, but nothing delivers them: "
"vanguard-alerts.sh and the deadman write to the journal and exit non-zero. Set ALERT_WEBHOOK_URL in .env."))
print(f"== diag : https://{diag_host}/ (Kibana: logs, dashboards, alerting; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag : (not set — run --full)")
print(f"== bot web : https://{bot_host}/ (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web : (not set — run --full)")
print(f"== gitea : https://{git_host}/ ssh port 2222 admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')} (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea : (not set — run --full)")
@@ -765,6 +951,11 @@ def main():
streamers, accounts = read_streamers(), read_accounts()
admin_pw = accounts.pop("admin", "")
editors = accounts
if mode == "--es-security":
# Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an
# --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but
# publishes nothing — three confusing symptoms with one cause.
es_security(compose_cmd()); return
if mode == "--show":
show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return
if mode == "--apply":

37
diag/index.html
View File

@@ -1,37 +0,0 @@
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>Vanguard diagnostics</title>
<style>
:root{--bg:#121417;--panel:#1c2026;--line:#2b313a;--fg:#f3f1ea;--mut:#a3acb8;--acc:#e5484d}
*{box-sizing:border-box}
body{background:var(--bg);color:var(--fg);font:16px/1.55 system-ui,-apple-system,"Segoe UI",sans-serif;margin:0;padding:48px 20px}
main{max-width:680px;margin:0 auto}
h1{font-size:30px;margin:0 0 6px;letter-spacing:.3px}
.lead{color:var(--mut);margin:0 0 30px}
a.card{display:block;padding:15px 18px;margin:0 0 10px;background:var(--panel);border:1px solid var(--line);
border-radius:12px;color:var(--fg);text-decoration:none}
a.card:hover{border-color:var(--acc)}
a.card b{display:block;font-size:17px}
a.card span{color:var(--mut);font-size:14px}
.note{margin-top:30px;padding:16px 18px;background:var(--panel);border:1px solid var(--line);border-radius:12px;
color:var(--mut);font-size:14px}
.note b{color:var(--fg)}
footer{margin-top:26px;color:#6b7280;font-size:13px}
footer a{color:#6b7280}
</style></head>
<body><main>
<h1>Vanguard diagnostics</h1>
<p class="lead">Logs, metrics and alerts for this server. Visible to <code>media-admins</code>.</p>
<a class="card" href="/logs"><b>Logs</b><span>every container and our own timers, last 30 days · VictoriaLogs</span></a>
<a class="card" href="/metrics-db"><b>Metrics</b><span>host, containers, MediaMTX, Caddy, last 90 days · VictoriaMetrics</span></a>
<a class="card" href="/alerts"><b>Alerts</b><span>what is firing now, and silences · Alertmanager</span></a>
<a class="card" href="/rules-metrics"><b>Metric rules</b><span>disk, load, memory, RAID, container restart loops</span></a>
<a class="card" href="/rules-logs"><b>Log rules</b><span>sweep failures, publish denials, failed bot releases</span></a>
<div class="note"><b>A deadman runs on the host, outside all of this.</b> Every five minutes it checks that each
service here still answers, that the alerting path is still delivering, and that the disks are not filling, and
it reports to Discord itself. Nothing on this page can tell you it has died, which is the point.</div>
<footer>Vanguard Media · <a href="/logout">sign out</a></footer>
</main></body></html>

50
docker-compose.yml
View File

@@ -72,7 +72,6 @@ services:
- ./dock:/srv/dock:ro
- ./clips:/srv/clips:ro # clip-intake self-service page
- ./about:/srv/about:ro # gateway page (/about)
- ./diag:/srv/diag:ro # diagnostics index (diag.HOST)
- /srv/live:/srv/live:ro # live portal static build (deployLive.sh)
- caddy_data:/data
- caddy_config:/config
@@ -235,6 +234,17 @@ services:
- ${BOT_APP_DIR:-/srv/bot/app}:/app # CI bundle installed by ./deployBotBundle.sh (code + vendor/)
- /srv/media/clips:/media/clips # rw: for a clip-submissions feature (writes into clips/<user>/)
- /srv/appdata/discord-bot:/data # persistent scratch (sqlite, caches) if the bot wants files
# Asks the BOT about itself rather than just proving the port accepts a connection: /health answers 503
# unless the Discord gateway is connected AND the database responds, and -f turns that into an unhealthy
# container, which Metricbeat's docker healthcheck metricset turns into an alarm. Requires the bot release
# that made /health unauthenticated and status-bearing — on older code every check 401s and -f fails,
# so ship the bot first. start_period covers command sync and the first roster load.
healthcheck:
test: ["CMD-SHELL", "curl -fsS -m 5 http://127.0.0.1:8000/health >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 120s
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -330,7 +340,18 @@ services:
# machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics.
ES_JAVA_OPTS: -Xms2g -Xmx2g
discovery.type: single-node
xpack.security.enabled: "false" # loopback only, behind Caddy + Authentik; no second login
# ON. Kibana's Observability apps (Infrastructure/Metrics) ask Elasticsearch whether the current user
# holds a privilege; with security off that API does not exist and they fail on every view. Humans still
# never see a password: Kibana signs everyone in as one user (observability/kibana.yml). Caddy +
# Authentik + DIAG_REQUIRE_GROUP remains the only real gate.
xpack.security.enabled: "true"
xpack.security.http.ssl.enabled: "false" # plain HTTP, published on loopback only
xpack.security.transport.ssl.enabled: "false" # single-node: no transport traffic, and the check is skipped
xpack.security.enrollment.enabled: "false"
xpack.security.autoconfiguration.enabled: "false"
# Bootstraps the elastic superuser on first start with security on. configure.py generates it and then
# sets up kibana_system and the service users; the error below means .env predates that.
ELASTIC_PASSWORD: "${ES_ELASTIC_PASSWORD:?run ./configure.py --apply first — it generates the Elastic passwords}"
xpack.ml.enabled: "false" # not used, and it reserves memory
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
@@ -344,7 +365,7 @@ services:
volumes:
- /srv/appdata/elasticsearch:/usr/share/elasticsearch/data
healthcheck:
test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
interval: 20s
timeout: 10s
retries: 12
@@ -356,14 +377,16 @@ services:
labels: { vanguard.service: kibana, vanguard.category: infra }
image: docker.elastic.co/kibana/kibana:9.2.0
container_name: kibana
# Everything non-secret lives in observability/kibana.yml (which explains the auth model); only the
# generated secrets arrive as environment variables, which that file interpolates.
environment:
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
SERVER_PUBLICBASEURL: https://${DIAG_HOST:-diag.example.com}
SERVER_BASEPATH: "" # served at the root of diag.HOST, so no prefix rewriting to get wrong
TELEMETRY_OPTIN: "false"
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
XPACK_REPORTING_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
XPACK_SECURITY_ENCRYPTIONKEY: ${KIBANA_ENCRYPTION_KEY}
DIAG_HOST: ${DIAG_HOST:-diag.example.com}
KIBANA_ENCRYPTION_KEY: ${KIBANA_ENCRYPTION_KEY}
ES_KIBANA_PASSWORD: "${ES_KIBANA_PASSWORD:?run ./configure.py --apply first}"
VG_DIAG_USER: ${VG_DIAG_USER:-vanguard_diag}
VG_DIAG_PASSWORD: "${VG_DIAG_PASSWORD:?run ./configure.py --apply first}"
volumes:
- ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
ports:
- "127.0.0.1:5601:5601"
depends_on:
@@ -375,6 +398,9 @@ services:
labels: { vanguard.service: filebeat, vanguard.category: infra }
image: docker.elastic.co/beats/filebeat:9.2.0
container_name: filebeat
environment: # Elasticsearch requires auth now; the config interpolates these
ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
user: root
command: ["filebeat", "-e", "--strict.perms=false"]
volumes:
@@ -382,6 +408,7 @@ services:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock # autodiscover reads container names and our vanguard.* labels
- /var/log/journal:/var/log/journal:ro # our own systemd units
- /run/log/journal:/run/log/journal:ro # ...and the volatile journal, if this host has no persistent one
- /etc/machine-id:/etc/machine-id:ro
- filebeat_data:/usr/share/filebeat/data
depends_on:
@@ -393,6 +420,9 @@ services:
labels: { vanguard.service: metricbeat, vanguard.category: infra }
image: docker.elastic.co/beats/metricbeat:9.2.0
container_name: metricbeat
environment: # Elasticsearch requires auth now; the config interpolates these
ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
user: root
command: ["metricbeat", "-e", "--strict.perms=false"]
# system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards.

11
install.sh
View File

@@ -38,6 +38,7 @@ echo "== media user + folders"
getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
usermod -aG docker "$OWNER" 2>/dev/null || true
install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
install -d -o 1000 -g 1000 /srv/media/.markers/metrics # script counters -> dumpd /metrics
install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
@@ -56,19 +57,21 @@ done
echo "== host scripts + units"
install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
install -m 0644 bin/vgmetrics.sh /usr/local/bin/ # sourced by the sweep/vgdeband; counters for dumpd's /metrics
install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot
install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it
# Compose bind-mounts this as a FILE; without it Docker would create a directory and alertmanager would not
# start. configure.py rewrites it from ALERT_WEBHOOK_URL on every apply; an empty one means "log only".
install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord
# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
if [[ ! -f /etc/vanguard/alert-webhook ]]; then
install -d -m 0750 /etc/vanguard
install -m 0640 -o root -g 65534 /dev/null /etc/vanguard/alert-webhook
install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook # root timers are the only readers
fi
install -d -m 0750 /etc/vanguard
install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer
systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
echo "== sysctl"
cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'

47
observability/filebeat.yml
View File

@@ -2,6 +2,11 @@
# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
# needs no parsing. See LOGGING.md for what each service's lines look like.
#
# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and
# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the
# process up with zero harvesters. Container logs are read by filestream + the container parser, and every
# filestream input needs its own stable id.
filebeat.autodiscover:
providers:
@@ -10,8 +15,20 @@ filebeat.autodiscover:
templates:
- condition: ~ # every container
config:
- type: container
- type: filestream
id: container-${data.docker.container.id}
paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
parsers:
- container: ~
prospector.scanner.symlinks: true
# Stamp the id here rather than letting add_docker_metadata guess it from the path.
processors:
- add_fields:
target: container
fields:
id: "${data.docker.container.id}"
name: "${data.docker.container.name}"
image.name: "${data.docker.container.image}"
filebeat.inputs:
- type: journald
@@ -25,7 +42,8 @@ filebeat.inputs:
- "_SYSTEMD_UNIT=vanguard-deadman.service"
processors:
- add_docker_metadata: ~
- add_docker_metadata:
match_fields: ["container.id"] # the id the autodiscover template stamped above
- add_host_metadata: ~
# promote our labels to real fields so Kibana filters on service.name and vanguard.category
- copy_fields:
@@ -36,15 +54,36 @@ processors:
to: vanguard.category
fail_on_error: false
ignore_missing: true
# The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in
# a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's
# vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target
# that already exists, so the first rule that matches wins and the other is skipped.
- copy_fields:
fields:
- from: container.labels.vanguard_service
to: event.dataset
- from: systemd.unit
to: event.dataset
fail_on_error: false
ignore_missing: true
- drop_fields:
fields: ["agent.ephemeral_id", "ecs.version", "input.type", "log.offset"]
fields: ["agent.ephemeral_id", "ecs.version", "log.offset"]
ignore_missing: true
output.elasticsearch:
hosts: ["http://elasticsearch:9200"]
# Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
# come from the container environment, which compose fills from .env — no secret is committed here.
username: "${ES_BEATS_USER}"
password: "${ES_BEATS_PASSWORD}"
setup.kibana:
setup.kibana: # no credentials: Kibana's anonymous provider signs these in
host: "http://kibana:5601"
setup.ilm.enabled: true # retention: set the delete phase once in Kibana (README 5h)
# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
setup.template.overwrite: true
setup.template.settings:
index.number_of_replicas: 0
logging.level: info

150
observability/kibana-alarms.py Executable file
View File

@@ -0,0 +1,150 @@
#!/usr/bin/env python3
"""The real alarm rules. Idempotent: creates what is missing, leaves what exists.
Every rule is an Elasticsearch query rule that counts matching documents in a window and writes one document
into the alert bus through the "Vanguard alert index" connector, which bin/vanguard-alerts.sh delivers to
Discord. Run observability/kibana-setup.py first — it creates that connector.
FIRING CADENCE: notify_when=onThrottleInterval with throttle=12h, so an alarm that stays true writes again
every 12 hours and stops when the condition clears. The delivery script's own 15-minute per-rule suppression
sits well inside that and never interferes.
WHAT IS DELIBERATELY NOT HERE
Elasticsearch, Kibana and Beat liveness, and the disks, are bin/vanguard-deadman.sh's job. Nothing that
depends on Kibana being alive can report that Kibana is dead, so those checks stay outside the stack and
duplicating them here would be worse than useless: they would go quiet in exactly the case that matters.
pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
-H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
KEY="$KEY" python3 observability/kibana-alarms.py
To change a rule, edit it in Kibana or delete it there and re-run this. Thresholds here are the defaults:
FREE_PCT 15 alarm when a real filesystem or memory has less than this percent free
STUCK_SECONDS 3600 alarm when a finished recording has waited this long to be normalised
THROTTLE 12h how often an unresolved alarm repeats
"""
import json, os, urllib.error, urllib.request
KIBANA = "http://127.0.0.1:5601"
HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
"Authorization": "ApiKey " + os.environ["KEY"]}
CONNECTOR_NAME = "Vanguard alert index"
FREE_PCT = float(os.environ.get("FREE_PCT", "15"))
STUCK_SECONDS = int(os.environ.get("STUCK_SECONDS", "3600"))
THROTTLE = os.environ.get("THROTTLE", "12h")
USED_MAX = round(1 - FREE_PCT / 100, 4) # 15% free -> used.pct >= 0.85
# Filesystems that are not disks. Without this, every tmpfs and overlay mount reports ~100% used and the
# disk alarm would fire permanently on something that cannot fill up.
PSEUDO_FS = ["tmpfs", "devtmpfs", "overlay", "squashfs", "ramfs", "autofs", "nsfs", "tracefs", "debugfs",
"configfs", "mqueue", "hugetlbfs", "pstore", "bpf", "devpts", "proc", "sysfs", "cgroup",
"cgroup2", "fusectl", "securityfs", "efivarfs", "binfmt_misc", "fuse.lxcfs", "iso9660"]
def rule(name, level, why, query, *, index="metricbeat-*", window=10, comparator=">", threshold=0):
return {"name": name, "level": level, "why": why, "query": query,
"index": index, "window": window, "comparator": comparator, "threshold": threshold}
ALARMS = [
rule("Disk free below %g%%" % FREE_PCT, "error",
"A filesystem is nearly full. Elasticsearch turns its indices READ-ONLY at the flood watermark and "
"does not undo that by itself, and recordings stop being published.",
{"query": {"bool": {
"filter": [{"range": {"system.filesystem.used.pct": {"gte": USED_MAX}}}],
"must_not": [{"terms": {"system.filesystem.type": PSEUDO_FS}}]}}}),
rule("Memory free below %g%%" % FREE_PCT, "warn",
"Memory is nearly exhausted. Elasticsearch has a pinned 2 GB heap, so the pressure is usually "
"ffmpeg or Jellyfin, and the kernel will start killing processes rather than queueing.",
{"query": {"range": {"system.memory.actual.used.pct": {"gte": USED_MAX}}}}),
rule("Discord bot not running", "error",
"No metrics from the discord-bot container for 5 minutes: it is stopped, removed, or restarting too "
"fast to be sampled. Nothing it does is happening.",
{"query": {"term": {"container.name": "discord-bot"}}},
window=5, comparator="<", threshold=1),
rule("Discord bot unhealthy", "error",
"The bot is running but reports itself degraded: its /health says the Discord gateway is "
"disconnected or the database does not answer. It is up and doing nothing.",
{"query": {"bool": {"filter": [{"term": {"container.name": "discord-bot"}},
{"term": {"docker.healthcheck.status": "unhealthy"}}]}}},
window=5),
rule("Recordings stuck waiting to be normalised", "warn",
"A finished recording has waited over %d minutes in .live/. It is not in Jellyfin, not in copyparty "
"and not announced, so to everyone else it does not exist." % (STUCK_SECONDS // 60),
{"query": {"range": {"prometheus.vanguard_media_pending_oldest_seconds.value": {"gt": STUCK_SECONDS}}}}),
rule("Normalise failures parked", "warn",
"One or more files have a normalize-failed log, so the sweep skips them forever until it is deleted. "
"The log now contains ffmpeg's stderr, which says why.",
{"query": {"range": {"prometheus.vanguard_media_failed_files.value": {"gt": 0}}}}),
]
def req(method, path, body=None):
request = urllib.request.Request(KIBANA + path, method=method,
data=None if body is None else json.dumps(body).encode())
for key, value in HEADERS.items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=30) as response:
raw = response.read()
return json.loads(raw) if raw else {}
except urllib.error.HTTPError as error:
print(f" {method} {path} -> HTTP {error.code}")
print(" " + error.read().decode(errors="replace")[:600])
raise SystemExit(1)
connectors = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
if CONNECTOR_NAME not in connectors:
raise SystemExit(f"connector {CONNECTOR_NAME!r} is missing — run observability/kibana-setup.py first")
connector_id = connectors[CONNECTOR_NAME]
present = {r["name"] for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])}
created = skipped = 0
for alarm in ALARMS:
if alarm["name"] in present:
print("already present:", alarm["name"])
skipped += 1
continue
made = req("POST", "/api/alerting/rule", {
"name": alarm["name"],
"rule_type_id": ".es-query",
"consumer": "stackAlerts",
"enabled": True,
"schedule": {"interval": "1m"},
"params": {
"searchType": "esQuery",
"index": [alarm["index"]],
"timeField": "@timestamp",
"esQuery": json.dumps(alarm["query"]),
"size": 1,
"aggType": "count",
"groupBy": "all",
"thresholdComparator": alarm["comparator"],
"threshold": [alarm["threshold"]],
"timeWindowSize": alarm["window"],
"timeWindowUnit": "m",
"excludeHitsFromPreviousRun": False,
},
"actions": [{
"group": "query matched",
"id": connector_id,
"params": {"documents": [{
"kind": "alarm",
"rule": "{{rule.name}}",
"level": alarm["level"],
"message": alarm["why"] + " [{{context.message}}]",
}]},
# every 12 h for as long as it stays true, then silence when it clears
"frequency": {"summary": False, "notify_when": "onThrottleInterval", "throttle": THROTTLE},
}],
})
print("created:", made["name"], "|", alarm["level"], "| repeats every", THROTTLE)
created += 1
print(f"\n{created} created, {skipped} already there. Disk/memory threshold {FREE_PCT:g}% free; "
f"stuck after {STUCK_SECONDS}s.")

161
observability/kibana-setup.py Executable file
View File

@@ -0,0 +1,161 @@
#!/usr/bin/env python3
"""Kibana first-run state. Idempotent; safe to re-run. configure.py --apply runs this for you.
connector "Vanguard alert index" — the .index connector, writing into the vanguard-alerts index
rule "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
data view "Vanguard logs + metrics" — filebeat-*,metricbeat-* so Discover and Lens have one source
WHY AN INDEX AND NOT A WEBHOOK
On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
leaves the stack (webhook, Slack, email, PagerDuty…) is Gold or above and returns
403 Action type .webhook is disabled because your basic license does not support it
So Kibana cannot talk to Discord. Instead every rule writes a document into ONE index, which is the alert
bus, and bin/vanguard-alerts.sh (a host timer) delivers from there to Discord and later to the bot. That is
strictly simpler than the webhook route: no Docker-bridge hop, no dumpd endpoint, and alerts are queryable
and replayable after the fact because they are just documents.
THE HEARTBEAT
bin/vanguard-deadman.sh alarms when no heartbeat document has arrived recently. It is the only check that
catches the ALERTING pipeline itself dying — if Kibana, its rule engine or the connector stops, no ordinary
alert can report it, because the thing that would report it is what broke. The rule queries metricbeat-*,
which always has recent documents, so it doubles as a Metricbeat liveness check.
Run it with an Elasticsearch API key: Basic is not an accepted scheme for this Kibana (the anonymous provider
is the only one enabled, so only ApiKey and Bearer are taken).
pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
-H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
KEY="$KEY" python3 observability/kibana-setup.py
Order matters: run this AFTER security is on. A rule created while security was off holds no API key and
stops working the moment it is enabled, until it is disabled and re-enabled.
ADDING A REAL ALARM: author it in Kibana as usual, point its action at the "Vanguard alert index" connector,
and give the document a message plus a level (warn|error). Do NOT set kind=heartbeat — that value is how the
delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
for real alarms; only the heartbeat wants to fire every interval.
"""
import json, os, time, urllib.error, urllib.request
KIBANA = "http://127.0.0.1:5601"
ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
"Authorization": "ApiKey " + os.environ["KEY"]}
def wait_for_kibana(seconds=300):
"""Kibana has to be *available*, not merely listening.
configure.py --es-security restarts Kibana as its last step, so anything that calls the Kibana API
immediately afterwards races its boot and gets a connection reset. Kibana also answers /api/status while
still starting, so the status level is what has to be checked, not the port.
"""
deadline = time.time() + seconds
announced = False
while time.time() < deadline:
try:
with urllib.request.urlopen(KIBANA + "/api/status", timeout=10) as response:
level = json.load(response).get("status", {}).get("overall", {}).get("level")
if level == "available":
print("kibana: available")
return
state = level or "no level"
except Exception as error:
state = type(error).__name__
if not announced:
print(f"kibana: waiting for it to become available ({state})…")
announced = True
time.sleep(5)
raise SystemExit(f"kibana did not become available within {seconds}s — nothing was created")
def req(method, path, body=None):
request = urllib.request.Request(KIBANA + path, method=method,
data=None if body is None else json.dumps(body).encode())
for key, value in HEADERS.items():
request.add_header(key, value)
try:
with urllib.request.urlopen(request, timeout=30) as response:
raw = response.read()
return json.loads(raw) if raw else {}
except urllib.error.HTTPError as error:
print(f" {method} {path} -> HTTP {error.code}")
print(" " + error.read().decode(errors="replace")[:600])
raise SystemExit(1)
wait_for_kibana()
# 1. the alert bus connector
CONNECTOR_NAME = "Vanguard alert index"
existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
if CONNECTOR_NAME in existing:
connector_id = existing[CONNECTOR_NAME]
print("connector already present:", connector_id)
else:
connector_id = req("POST", "/api/actions/connector", {
"name": CONNECTOR_NAME,
"connector_type_id": ".index",
# executionTimeField stamps every document, so a rule body never has to get the timestamp right.
"config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
"secrets": {}})["id"]
print("connector created:", connector_id, "->", ALERT_INDEX)
# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
# harmless and meant the data view was never created on any system where the rule already existed, which
# is every re-run.
RULE_NAME = "Vanguard alerting heartbeat"
existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
if r["name"] == RULE_NAME), None)
if existing_rule:
print("rule already present:", existing_rule["id"],
"enabled" if existing_rule.get("enabled") else "DISABLED")
else:
rule = req("POST", "/api/alerting/rule", {
"name": RULE_NAME,
"rule_type_id": ".es-query",
"consumer": "stackAlerts",
"enabled": True,
"schedule": {"interval": "1m"},
"params": {
"searchType": "esQuery",
"index": ["metricbeat-*"],
"timeField": "@timestamp",
"esQuery": json.dumps({"query": {"match_all": {}}}),
"size": 1,
"aggType": "count",
"groupBy": "all",
"thresholdComparator": ">",
"threshold": [0],
"timeWindowSize": 5,
"timeWindowUnit": "m",
# re-evaluate the whole window every run, so the rule stays active instead of going quiet once
# it has already seen those documents
"excludeHitsFromPreviousRun": False,
},
# onActiveAlert writes on EVERY check. Kibana's default only acts when a rule CHANGES status, which
# would produce one heartbeat and then silence — the deadman would read that as a dead pipeline.
"actions": [{
"group": "query matched",
"id": connector_id,
"params": {"documents": [{
"kind": "heartbeat", # the delivery script drops these; never reach Discord
"rule": "{{rule.name}}",
"level": "info",
"message": "kibana rule engine and the index connector are alive",
}]},
"frequency": {"summary": False, "notify_when": "onActiveAlert", "throttle": None},
}],
})
print("rule created:", rule["id"], "| enabled:", rule.get("enabled"))
# 3. the data view. Without one, Discover and Lens have nothing to point at and every panel starts with a
# detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
# metrics, which is the whole reason they share a cluster.
VIEW_TITLE = "filebeat-*,metricbeat-*"
views = req("GET", "/api/data_views").get("data_view", [])
if any(v.get("title") == VIEW_TITLE for v in views):
print("data view already present:", VIEW_TITLE)
else:
view = req("POST", "/api/data_views/data_view", {
"data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)

41
observability/kibana.yml Normal file
View File

@@ -0,0 +1,41 @@
# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
# into .env, so this file is committed and contains none.
#
# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
# whether the current user holds a privilege. With security off that API does not exist, and the
# Infrastructure / Metrics app fails on every page view with
# no handler found for uri [/_security/user/_has_privileges]
# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
#
# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
# scheme and have Caddy inject a key chosen by Authentik group).
server.host: "0.0.0.0"
server.publicBaseUrl: "https://${DIAG_HOST}"
# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
telemetry.optIn: false
elasticsearch.hosts: ["http://elasticsearch:9200"]
elasticsearch.username: "kibana_system" # Kibana's own internal account, NOT what visitors become
elasticsearch.password: "${ES_KIBANA_PASSWORD}"
# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
# The only provider, so there is no login page and no login selector: every request is already signed in.
xpack.security.authc.providers:
anonymous.anonymous1:
order: 0
credentials:
username: "${VG_DIAG_USER}"
password: "${VG_DIAG_PASSWORD}"

20
observability/metricbeat.yml
View File

@@ -27,6 +27,15 @@ metricbeat.modules:
metrics_path: /metrics
hosts: ["http://127.0.0.1:2019"]
- module: prometheus # dumpd: OUR metrics — normalise queue, encode time, live streams
period: 30s
metrics_path: /metrics
hosts: ["http://127.0.0.1:8080"] # dumpd's loopback listener; Caddy routes no host to /metrics
# These are real counters, so let them arrive typed: Kibana then gets .rate (per-second) as well as the
# raw total, which is what makes "encode seconds per hour" a chart rather than a staircase.
use_types: true
rate_counters: true
processors:
- add_host_metadata: ~
# The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a
@@ -43,9 +52,18 @@ processors:
output.elasticsearch:
hosts: ["http://127.0.0.1:9200"]
# Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
# come from the container environment, which compose fills from .env — no secret is committed here.
username: "${ES_BEATS_USER}"
password: "${ES_BEATS_PASSWORD}"
setup.kibana:
setup.kibana: # no credentials: Kibana's anonymous provider signs these in
host: "http://127.0.0.1:5601"
setup.dashboards.enabled: true # the system and docker dashboards, so there is something to look at
# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
setup.template.overwrite: true
setup.template.settings:
index.number_of_replicas: 0
logging.level: info

9
systemd/vanguard-alerts.service Normal file
View File

@@ -0,0 +1,9 @@
[Unit]
Description=Vanguard alerts: deliver Kibana's alert documents to Discord
After=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/vanguard-alerts.sh
# Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts
# index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md.

10
systemd/vanguard-alerts.timer Normal file
View File

@@ -0,0 +1,10 @@
[Unit]
Description=Deliver Vanguard alerts every minute
[Timer]
OnBootSec=3min
OnUnitActiveSec=1min
AccuracySec=15s
[Install]
WantedBy=timers.target