techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3662eb3465b651425c656fd85e30de31cd54ac41
Media-Server-Setup/LOGGING.md
Brandyn / Techy 3662eb3465 Phase 14: diag. host with VictoriaLogs at /logs, ops controls later 
Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-03 15:33:47 -06:00

6.8 KiB
Raw Blame History

Logging framework

Decided 2026-11-01, and the store chosen on 2026-11-03: VictoriaLogs, on the diag.BASE host with its UI at /logs (Phase 14 in CommunityDiscordMedia.md). Everything upstream of the store was fixed first, so adopting it is a collector config rather than a code change.

Where the lines come from (the only two taps a collector needs)

Tap Holds Read with
journald our systemd units: dumpd, normalize-sweep, trash-purge, deploy-pending, vgdeband (transient), deploy.sh output journalctl -u <unit>; collectors: Alloy loki.source.journal, Vector journald, VictoriaLogs journald ingestion, Netdata's journal plugin
Docker json-file logs every container (docker-compose.yml), rotated 5 × 50 MB each docker logs <name>; collectors read /var/lib/docker/containers/*/*-json.log and get the container labels below for free

Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback.

Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata)

endpoint what from
http://127.0.0.1:9100/metrics host: CPU, RAM, disk space + I/O, network, load, md RAID node_exporter (host network, pid host)
http://127.0.0.1:9101/metrics per container: CPU, memory, network, block I/O — labelled with vanguard.service / vanguard.category cAdvisor
http://127.0.0.1:9998/metrics MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets MediaMTX metrics: yes
http://127.0.0.1:2019/metrics Caddy: requests by host/status, durations, TLS Caddy admin API (servers { metrics })

None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with: disk on /srv > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.

Container labels (set in docker-compose.yml, x-logging anchor + labels:)

vanguard.service = the service name, vanguard.category = one of the categories below. A collector turns these into stream labels without parsing a single line.

Categories = the questions people ask

category services / components correlation key
stream mediamtx (path ready/closed, recorder, WebRTC sessions, the on-ready hook's event=mirror), dumpd publish_allow / publish_deny, sweep event=published stream = user/Title
auth caddy access logs (status, X-Authentik-Username), authentik-server/worker events, gitea OIDC user
media sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty file
intake syncthing (folder.id), dumpd clips_setup, quota_* folder = clips-<user>
infra mariadb, caddy TLS/ACME, container restarts, disk host only
deploy deploy.sh, dumpd deploy_*, bot_deploy_* and bot_env_* (the bot's CI posting a release bundle and its settings), act_runner rev
code, bot gitea, discord-bot —

Labels vs fields

  • Labels (low cardinality, what you filter by): service, category, level, unit. A dozen values each.
  • Fields (parsed at query time, what you search in): stream, user, file, folder, session, status, ip, event. Thousands of values — fine as fields, ruinous as labels (Loki). VictoriaLogs indexes every word anyway.

Our own line format: logfmt, one event per line

ts=2026-11-01T19:52:14-06:00 level=info component=sweep event=published stream="techy2493/Vanguard_Training" file="/srv/media/streams/…" verdict=encode
  • ts ISO-8601 with offset · level info|warn|error · component dumpd|sweep|vgdeband|deploy|on-ready · event a verb/noun in snake_case · then key=value, values with spaces or quotes double-quoted.
  • Same key means the same thing everywhere: stream (user/Title), user, file, folder, rev, ip, reason.
  • Implemented by: log() in bin/dumpd.py, vglog in bin/normalize-sweep.sh and bin/vgdeband.sh, log() in deploy.sh (msg= only), bin/mtx-on-ready.sh (MediaMTX prefixes its own path/time).

Events today: dumpd publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused bot_env_applied bot_env_failed; sweep verdict timecode deband fill_gaps published done failed skip; vgdeband deband done failed skip; hook mirror. A future alert script watches level=error, event=failed, event=publish_deny bursts, event=quota_paused, plus disk.

Third-party formats a collector must parse

source format stage
caddy JSON (logger, level, msg; access logs: request.*, status, request.headers.X-Authentik-Username) json
authentik JSON (event, user, action) json
syncthing key=value after a date time: prefix (folder.id=… log.pkg=model) logfmt
mediamtx `YYYY/MM/DD HH:MM:SS LVL [component] [session path x] message`
jellyfin [HH:MM:SS.mmm] [LVL] [thread] Logger.Class: message + stack-trace continuation lines multiline (join lines not starting with [) then regex
gitea, mariadb, copyparty plain text level regex only

Caddy access logs

On for every site block (CADDY_LOG in configure.py): JSON to stderr; Cookie, Authorization and X-Stream-Auth request headers are deleted before logging, the X-Authentik-* identity headers are kept — that is the "who downloaded what" audit trail for the files host.

Retention

One global retention period in VictoriaLogs (-retentionPeriod); start at 30 days and watch the disk. Keeping auth and deploy longer than the rest would need separate tenants, which is not worth it unless an audit does.

Wiring sketches (not deployed; pick one later)

  • Grafana + Loki: alloy container with loki.source.journal + loki.source.docker (labels from the container labels above) → loki on loopback → grafana behind Authentik OIDC (third provider, like Jellyfin/Gitea). Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
  • VictoriaLogs (chosen): single container on 127.0.0.1:9428, run with -http.pathPrefix=/logs; Vector ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on diag.BASE (media-admins). VictoriaMetrics beside it would consume the four metric taps.
  • Netdata: single container, host + container metrics and alerts (Discord) out of the box, journald viewer; behind the Caddy gate. Weakest at log search.
  • ELK: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a single host's logs.