Logging framework: logfmt events in our scripts, container labels + rotated json-file logs, Caddy JSON access logs; LOGGING.md

Store-agnostic: journald + Docker logs are the only taps any collector (Alloy/Loki, VictoriaLogs, Netdata) needs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

LOGGING.md

@@ -0,0 +1,84 @@
+# Logging framework — store-agnostic
+
+Decided 2026-11-01. The log *store* (Grafana + Loki, VictoriaLogs, Netdata, …) is not chosen yet. Everything
+upstream of it is fixed now, so the choice later is a collector config, not a code change.
+
+## Where the lines come from (the only two taps a collector needs)
+
+| Tap | Holds | Read with |
+|---|---|---|
+| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
+| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
+
+Every collector on the shortlist reads both. Metrics are the same story: node_exporter / cAdvisor expose the
+Prometheus text format that Alloy, vmagent and Netdata all scrape; MediaMTX (`metrics: yes`) and Caddy have
+endpoints too, currently off.
+
+## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
+
+`vanguard.service` = the service name, `vanguard.category` = one of the categories below. A collector turns these
+into stream labels without parsing a single line.
+
+## Categories = the questions people ask
+
+| category | services / components | correlation key |
+|---|---|---|
+| `stream` | mediamtx (path ready/closed, recorder, WebRTC sessions, the on-ready hook's `event=mirror`), dumpd `publish_allow` / `publish_deny`, sweep `event=published` | `stream` = `user/Title` |
+| `auth` | caddy access logs (status, `X-Authentik-Username`), authentik-server/worker events, gitea OIDC | `user` |
+| `media` | sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty | `file` |
+| `intake` | syncthing (`folder.id`), dumpd `clips_setup`, `quota_*` | `folder` = `clips-<user>` |
+| `infra` | mariadb, caddy TLS/ACME, container restarts, disk | host only |
+| `deploy` | deploy.sh, dumpd `deploy_*`, act_runner | `rev` |
+| `code`, `bot` | gitea, discord-bot | — |
+
+## Labels vs fields
+
+- **Labels** (low cardinality, what you filter by): `service`, `category`, `level`, `unit`. A dozen values each.
+- **Fields** (parsed at query time, what you search in): `stream`, `user`, `file`, `folder`, `session`, `status`, `ip`, `event`.
+  Thousands of values — fine as fields, ruinous as labels (Loki). VictoriaLogs indexes every word anyway.
+
+## Our own line format: logfmt, one event per line
+
+    ts=2026-11-01T19:52:14-06:00 level=info component=sweep event=published stream="techy2493/Vanguard_Training" file="/srv/media/streams/…" verdict=encode
+
+- `ts` ISO-8601 with offset · `level` info|warn|error · `component` dumpd|sweep|vgdeband|deploy|on-ready · `event` a verb/noun in snake_case · then `key=value`, values with spaces or quotes double-quoted.
+- Same key means the same thing everywhere: `stream` (user/Title), `user`, `file`, `folder`, `rev`, `ip`, `reason`.
+- Implemented by: `log()` in `bin/dumpd.py`, `vglog` in `bin/normalize-sweep.sh` and `bin/vgdeband.sh`, `log()` in `deploy.sh` (`msg=` only), `bin/mtx-on-ready.sh` (MediaMTX prefixes its own path/time).
+
+Events today: dumpd `publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_*`;
+sweep `verdict timecode deband fill_gaps published done failed skip`; vgdeband `deband done failed skip`; hook `mirror`.
+A future alert script watches `level=error`, `event=failed`, `event=publish_deny` bursts, `event=quota_paused`, plus disk.
+
+## Third-party formats a collector must parse
+
+| source | format | stage |
+|---|---|---|
+| caddy | JSON (`logger`, `level`, `msg`; access logs: `request.*`, `status`, `request.headers.X-Authentik-Username`) | json |
+| authentik | JSON (`event`, `user`, `action`) | json |
+| syncthing | key=value after a `date time:` prefix (`folder.id=… log.pkg=model`) | logfmt |
+| mediamtx | `YYYY/MM/DD HH:MM:SS LVL [component] [session|path x] message` | one regex → component, id, msg |
+| jellyfin | `[HH:MM:SS.mmm] [LVL] [thread] Logger.Class: message` + stack-trace continuation lines | **multiline** (join lines not starting with `[`) then regex |
+| gitea, mariadb, copyparty | plain text | level regex only |
+
+## Caddy access logs
+
+On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie`, `Authorization` and
+`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
+"who downloaded what" audit trail for the files host.
+
+## Retention (when a store exists)
+
+14 days for everything; 90 for `auth` and `deploy`. Both Loki and VictoriaLogs do per-stream retention.
+
+## Wiring sketches (not deployed; pick one later)
+
+- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
+  labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
+  Metrics: Alloy scrapes node_exporter, cAdvisor, MediaMTX `:9998/metrics`, Caddy `:2019/metrics` → Prometheus or
+  VictoriaMetrics, same Grafana.
+- **VictoriaLogs**: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy
+  forward-auth gate (`media-admins`). VictoriaMetrics beside it for metrics.
+- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
+  behind the Caddy gate. Weakest at log search.
+- **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a
+  single host's logs.