Media-Server-Setup/LOGGING.md

Logging framework — store-agnostic

Decided 2026-11-01. The log store (Grafana + Loki, VictoriaLogs, Netdata, …) is not chosen yet. Everything upstream of it is fixed now, so the choice later is a collector config, not a code change.

Where the lines come from (the only two taps a collector needs)

Tap	Holds	Read with
journald	our systemd units: dumpd, normalize-sweep, trash-purge, deploy-pending, vgdeband (transient), deploy.sh output	journalctl -u <unit>; collectors: Alloy loki.source.journal, Vector journald, VictoriaLogs journald ingestion, Netdata's journal plugin
Docker json-file logs	every container (docker-compose.yml), rotated 5 × 50 MB each	docker logs <name>; collectors read /var/lib/docker/containers/*/*-json.log and get the container labels below for free

Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback.

Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata)

endpoint	what	from
http://127.0.0.1:9100/metrics	host: CPU, RAM, disk space + I/O, network, load, md RAID	node_exporter (host network, pid host)
http://127.0.0.1:9101/metrics	per container: CPU, memory, network, block I/O — labelled with vanguard.service / vanguard.category	cAdvisor
http://127.0.0.1:9998/metrics	MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets	MediaMTX metrics: yes
http://127.0.0.1:2019/metrics	Caddy: requests by host/status, durations, TLS	Caddy admin API (servers { metrics })

None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with: disk on /srv > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.

Container labels (set in docker-compose.yml, x-logging anchor + labels:)

vanguard.service = the service name, vanguard.category = one of the categories below. A collector turns these into stream labels without parsing a single line.

Categories = the questions people ask

category	services / components	correlation key
stream	mediamtx (path ready/closed, recorder, WebRTC sessions, the on-ready hook's event=mirror), dumpd publish_allow / publish_deny, sweep event=published	stream = user/Title
auth	caddy access logs (status, X-Authentik-Username), authentik-server/worker events, gitea OIDC	user
media	sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty	file
intake	syncthing (folder.id), dumpd clips_setup, quota_*	folder = clips-<user>
infra	mariadb, caddy TLS/ACME, container restarts, disk	host only
deploy	deploy.sh, dumpd deploy_*, bot_deploy_* and bot_env_* (the bot's CI posting a release bundle and its settings), act_runner	rev
code, bot	gitea, discord-bot	—

Labels vs fields

• Labels (low cardinality, what you filter by): service, category, level, unit. A dozen values each.
• Fields (parsed at query time, what you search in): stream, user, file, folder, session, status, ip, event. Thousands of values — fine as fields, ruinous as labels (Loki). VictoriaLogs indexes every word anyway.

Our own line format: logfmt, one event per line

ts=2026-11-01T19:52:14-06:00 level=info component=sweep event=published stream="techy2493/Vanguard_Training" file="/srv/media/streams/…" verdict=encode

• ts ISO-8601 with offset · level info|warn|error · component dumpd|sweep|vgdeband|deploy|on-ready · event a verb/noun in snake_case · then key=value, values with spaces or quotes double-quoted.
• Same key means the same thing everywhere: stream (user/Title), user, file, folder, rev, ip, reason.
• Implemented by: log() in bin/dumpd.py, vglog in bin/normalize-sweep.sh and bin/vgdeband.sh, log() in deploy.sh (msg= only), bin/mtx-on-ready.sh (MediaMTX prefixes its own path/time).

Events today: dumpd publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused bot_env_applied bot_env_failed; sweep verdict timecode deband fill_gaps published done failed skip; vgdeband deband done failed skip; hook mirror. A future alert script watches level=error, event=failed, event=publish_deny bursts, event=quota_paused, plus disk.

Third-party formats a collector must parse

source	format	stage
caddy	JSON (logger, level, msg; access logs: request.*, status, request.headers.X-Authentik-Username)	json
authentik	JSON (event, user, action)	json
syncthing	key=value after a date time: prefix (folder.id=… log.pkg=model)	logfmt
mediamtx	`YYYY/MM/DD HH:MM:SS LVL [component] [session	path x] message`
jellyfin	[HH:MM:SS.mmm] [LVL] [thread] Logger.Class: message + stack-trace continuation lines	multiline (join lines not starting with [) then regex
gitea, mariadb, copyparty	plain text	level regex only

Caddy access logs

On for every site block (CADDY_LOG in configure.py): JSON to stderr; Cookie, Authorization and X-Stream-Auth request headers are deleted before logging, the X-Authentik-* identity headers are kept — that is the "who downloaded what" audit trail for the files host.

Retention (when a store exists)

14 days for everything; 90 for auth and deploy. Both Loki and VictoriaLogs do per-stream retention.

Wiring sketches (not deployed; pick one later)

• Grafana + Loki: alloy container with loki.source.journal + loki.source.docker (labels from the container labels above) → loki on loopback → grafana behind Authentik OIDC (third provider, like Jellyfin/Gitea). Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
• VictoriaLogs: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy forward-auth gate (media-admins). VictoriaMetrics beside it for metrics.
• Netdata: single container, host + container metrics and alerts (Discord) out of the box, journald viewer; behind the Caddy gate. Weakest at log search.
• ELK: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a single host's logs.