Sweep: crop the timecode band off published streams (re-encode after decode); vgdeband.sh catch-up

The published recording is the game picture + audio + tmcd track, nothing else. Streams without the
band are still remuxed losslessly. vgstrip plan exports TC_BAND and marks the sidecar band_removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -306,9 +306,10 @@ permanent reference is needed for long sessions, not just a start burst.
   frame-0 capture time = median(decoded − pts) at the start (frame-accurate) → QuickTime `tmcd` track
   (time of day, Eastern, file fps) + `creation_time` (UTC) in the existing lossless remux; a `.tc.json`
   sidecar with the samples records drift over the file. No strip → server arrival time (coarse), logged.
-  Never re-encode, never crop.
+  Strip present → the band is cropped off in a re-encode (decided 2026-11-01: finals = picture + audio + tmcd,
-- **Viewing**: live portal crops the band with CSS; Jellyfin shows a thin strip at the bottom edge (VHS-style).
+  nothing else); no strip → lossless remux, never crop.
-  Editors export at the game resolution (2560×1440), which drops the band.
+- **Viewing**: live portal crops the band on the live picture (canvas/overflow fix pending); recordings are
+  already at game resolution.
 - MediaMTX `recordPath` with `%f` (microseconds) for the fallback; plugin `ItemTimestamp` prefers `creation_time`.
 - [x] `/clock/` page: offset measurement, strip encoder, readable clock, band layout
 - [x] dock: overscan band setup + browser source via obs-websocket; opt-out

README.md

@@ -386,7 +386,7 @@ restarts only copyparty.
 ## Timecode for multicam editing (frame accurate)
 
 Every published stream carries a QuickTime **timecode track** (`tmcd`, time of day in Eastern, at the file's
-frame rate) and a UTC `creation_time`, written by the sweep during the lossless remux, plus a sidecar
+frame rate) and a UTC `creation_time`, written by the sweep when it publishes the file, plus a sidecar
 `<Title>__<ts>.json` with the exact capture time (the Jellyfin plugin reads it). Resolve and Premiere read the
 track natively: drop clips from several streamers on a multicam timeline, sync by timecode.
 
@@ -394,14 +394,19 @@ Where the time comes from, best first:
 1. **Overscan band** — a thin strip of extra rows below the picture carrying the *server* clock, encoded
    (black/white cells, format in `bin/vgstrip.py`) plus a small readable clock. The server decodes it
    (`bin/vgstrip.py`) at the start, middle and end of each recording → the timecode is frame-accurate at the
-   start and the sidecar records drift over the file. Game image untouched. OBS: dock → *Timecode band* →
+   start and the sidecar records drift over the file. **Then the band is cropped off**: the sweep re-encodes the
+   stream (libx264 CRF 18, frame timing preserved, audio copied) so the published file is the game image + audio +
+   the `tmcd` track and nothing else; the raw with the band is deleted (`KEEP_RAW_STREAMS=1` keeps it in `.src/`).
+   Streams without the band are remuxed losslessly. Cost: a 3 h 1440p60 night is a few hours of CPU on the
+   E-2136; the sweep is niced. OBS: dock → *Timecode band* →
    "Add band + clock source" (canvas grows by W/160 rows, a locked browser source `Vanguard Clock` →
    `https://live.BASE/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
 2. **Server arrival time** (no band) — microseconds in the recording filename. Within ~0.1–0.3 s per streamer
    (encoder + network delay); coarse but always available.
 
-Viewing: the live portal crops the band; Jellyfin shows it as a thin strip at the bottom (VHS-style);
+Viewing: the live portal crops the band on the live picture (the only place it still exists); recordings in
-**editors export at the game resolution (e.g. 2560×1440) which drops the band** — finals carry nothing.
+Jellyfin, copyparty and the editors' timelines are already at game resolution. Recordings published before
+the crop existed: `sudo vgdeband.sh [FILE…]` re-encodes them the same way and marks the sidecar.
 Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
 Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
 
@@ -528,7 +533,7 @@ upscaled; aspect kept).
 | input | action |
 |-------|--------|
 | already compliant | nothing |
-| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), lossless remux to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time`, write the `.json` sidecar, remove `.live` and the empty title folder |
+| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), then publish to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time` — band present: re-encode with the band cropped off; no band: lossless remux — write the `.json` sidecar, remove `.live` and the empty title folder |
 | in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
 | HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
 | ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |

STATUS.md

@@ -11,7 +11,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
 | Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
 | Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
-| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, lossless remux out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar, one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
+| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
 | Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
 | Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
 | Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
@@ -32,7 +32,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
 | Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
 | Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
-| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; export at the game resolution to drop the band | Resolve / Premiere | — |
+| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
 | Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
 | Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
 | Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
@@ -70,6 +70,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 
 ## Unfinished, in rough priority
 
+0a. **Band removal deploy** (committed 2026-11-01): install `bin/vgstrip.py`, `bin/normalize-sweep.sh`, `bin/vgdeband.sh`; run `vgdeband.sh` once on the one published recording that still has the band (techy2493 / Vanguard Training). Live portal: band still shown on the live picture → canvas positioning + overflow hide, later.
 0. **Hostname restructure deploy** (committed 2026-11-01): DNS CNAMEs `media`, `files`, `clips`, `sync`; `./configure.py --full` with `media.` as main; MediaMTX restart between streams; `./deployLive.sh`; streamers re-add the dock at `live./dock/` and redo "Add band + clock source"; re-pin Discord links. Procedure C in HOSTNAME-CHANGE.md.
 
 1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.

bin/normalize-sweep.sh

@@ -6,7 +6,10 @@
 # - anything else (HEVC, 10-bit, VFR, MKV, >1440p, >60fps): full-res re-encode -> name.mp4,
 #   raw source kept in a hidden .src/ subfolder next to it (Jellyfin/copyparty hide dot-dirs)
 # - streams/<user>/<Title>/.live/<ts>.mp4 (MediaMTX writes here, hidden from Jellyfin/copyparty while live):
-#   when finished -> flat MP4 published as streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
+#   when finished -> the timecode overscan band (if the streamer had it on) is decoded and then CROPPED OFF in a
+#   re-encode (libx264, frame timing preserved, audio copied) so the published file is the game image + audio +
+#   a tmcd timecode track, nothing else; streams without the band are remuxed losslessly. Published as flat
+#   MP4 streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
 #   in the streamer's folder: Jellyfin would merge several files inside one title folder into ONE episode
 #   with "versions"), the .live source deleted, the empty title folder removed.
 # - Syncthing folders (marker dir .stfolder in the folder root, receive-only): compliant files are left
@@ -23,6 +26,9 @@ MINS_IDLE=3
 # the total missing time is >= this many ms. 0 = never (default; a 3 h re-encode is ~6 h of CPU on the E-2136).
 # Set via systemd Environment=FILL_GAPS_MS=... ; viewers already see held frames either way.
 FILL_GAPS_MS="${FILL_GAPS_MS:-0}"
+# Band removal re-encodes every stream that carried the timecode band. The raw (with band) is deleted after a
+# successful encode; KEEP_RAW_STREAMS=1 keeps it in the hidden .src/ instead (doubles disk per night).
+KEEP_RAW_STREAMS="${KEEP_RAW_STREAMS:-0}"
 MAX_W=2560; MAX_H=1440; MAX_FPS=60
 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
 flock -n 9 || exit 0
@@ -110,7 +116,7 @@ while IFS= read -r -d '' src; do
   fi
   tmp="$out.part"
   log "$verdict  $src  ($reason)"
-  tcargs=(); tcjson=""
+  tcargs=(); tcjson=""; band=0; deband=0
   if (( live )); then
     # timecode: decode the overscan strip (server clock) or fall back to the server arrival time in the filename
     stem="$(basename "${src%.*}")"
@@ -132,6 +138,11 @@ PYEOF
       tcjson="${TC_JSON:-}"
       log "tc     $src  source=$TC_SOURCE timecode=$TC_TIMECODE"
     fi
+    # the band has done its job once decoded: remove it from the picture (re-encode with a crop)
+    if [[ ${TC_SOURCE:-} == strip && ${TC_BAND:-0} -gt 0 ]]; then
+      band=$TC_BAND; deband=1; verdict=encode; reason="remove ${band}px timecode band"
+      log "deband $src  cropping ${band} rows (re-encode, timing preserved)"
+    fi
     # opt-in gap filling: switch this stream from remux to a CFR re-encode when the holes add up to enough
     if (( FILL_GAPS_MS > 0 )) && [[ $verdict == remux && -n $tcjson ]]; then
       missing=$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get("gaps",{}).get("total_missing_ms",0))' "$tcjson" 2>/dev/null || echo 0)
@@ -146,13 +157,18 @@ PYEOF
   else
     # scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
     vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
+    (( deband )) && vf="crop=iw:ih-${band}:0:0,$vf"          # drop the band rows first (band is even, so sizes stay even)
     fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
     awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
+    # streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
+    fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
+    if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
+    if (( live )) && [[ $reason != *audio=* ]]; then aargs=(-c:a copy); fi
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
       -map 0:v:0 -map '0:a?' \
       -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high \
-      -vf "$vf" -fps_mode cfr "${fpsargs[@]}" \
+      -vf "$vf" -fps_mode "$fpsmode" "${fpsargs[@]}" \
-      -c:a aac -b:a 160k \
+      "${aargs[@]}" \
       "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
   fi
   rc=$?
@@ -163,11 +179,11 @@ PYEOF
       if [[ -n $tcjson ]]; then                        # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
         printf '%s\n' "$tcjson" > "${out%.*}.json"; chown 1000:1000 "${out%.*}.json" 2>/dev/null
       fi
-      if [[ $verdict == encode ]]; then                # gap-filled re-encode: keep the raw recording in the hidden .src/
+      if [[ $verdict == encode ]] && (( ! deband || KEEP_RAW_STREAMS )); then   # gap-filled re-encode (or asked to): keep the raw in the hidden .src/
         srcdir="$(dirname "$out")/.src"; mkdir -p "$srcdir"; chown 1000:1000 "$srcdir" 2>/dev/null
         mv -f "$src" "$srcdir/$(basename "$out" .mp4).raw.mp4"
       else
-        rm -f "$src"
+        rm -f "$src"                                   # band removal: the cropped file IS the recording; raw-with-band is not kept
       fi
       rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
       log "done   $out"; continue

bin/vgdeband.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# One-off / catch-up: remove the timecode overscan band from stream recordings that were published BEFORE the
+# sweep learned to crop it. Re-encodes (libx264, frame timing preserved, audio copied), re-applies the timecode
+# from the sidecar, replaces the file, marks the sidecar band_removed. Skips files whose sidecar already says so.
+#   vgdeband.sh                      all of /srv/media/streams/*/*.mp4 with a strip sidecar
+#   vgdeband.sh FILE.mp4 ...         just these
+# Long: a 3 h 1440p60 night is a few hours of CPU. Run detached: sudo systemd-run --unit vgdeband -p Nice=15 \
+#   -p IOSchedulingClass=idle /usr/local/bin/vgdeband.sh ; follow with journalctl -fu vgdeband
+set -uo pipefail
+ROOT="${MEDIA_ROOT:-/srv/media}"
+log() { echo "$(date '+%F %T') $*"; }
+dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
+if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi
+for f in "${files[@]}"; do
+  side="${f%.*}.json"
+  [[ -f $side ]] || { log "skip   $f (no sidecar)"; continue; }
+  read -r src removed tc ctime < <(python3 -c '
+import json,sys; d=json.load(open(sys.argv[1]))
+print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode",""), d.get("timestamp",""))' "$side")
+  [[ $src == strip ]] || { log "skip   $f (source=$src)"; continue; }
+  (( removed )) && { log "skip   $f (band already removed)"; continue; }
+  fuser -s "$f" 2>/dev/null && { log "skip   $f (in use)"; continue; }
+  w=$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=p=0 "$f")
+  band=$(python3 /usr/local/bin/vgstrip.py band "$w")
+  tmp="$f.part"
+  tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime")
+  log "deband $f  (${w}px wide -> crop ${band} rows; tc=$tc)"
+  nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \
+    -map 0:v:0 -map '0:a?' \
+    -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high \
+    -vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \
+    "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
+  rc=$?; d1=$(dur "$f"); d2=$(dur "$tmp")
+  if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
+    mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null
+    python3 -c '
+import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True
+json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null
+    log "done   $f"
+  else
+    rm -f "$tmp"; log "FAILED $f rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} (file left untouched)" >&2
+  fi
+done

bin/vgstrip.py

@@ -162,6 +162,11 @@ def plan(path, arrival_ms=None, tz="America/New_York"):
     else:
         t0 = None; src = "none"
     out = {"source": src, "fps": r.get("fps"), "drift_ms": r.get("drift_ms", 0), "samples": r.get("samples", [])}
+    if src == "strip":
+        # the band is removed (cropped off in a re-encode) by the sweep once it has been decoded; record it
+        _, band = geometry(r["width"]) if r.get("width") else geometry(probe(path)[0])
+        out.update({"band_rows": band, "band_removed": True})
+        print(f"TC_BAND={band}")
     try:
         out["gaps"] = find_gaps(path, r.get("fps") or 60.0)
     except Exception:
@@ -181,4 +186,6 @@ if __name__ == "__main__":
     if len(sys.argv) < 2: print(__doc__); sys.exit(2)
     if sys.argv[1] == "plan":
         plan(sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 and sys.argv[3] else None); sys.exit(0)
+    if sys.argv[1] == "band":                 # band rows for a canvas width (vgdeband.sh)
+        print(geometry(int(sys.argv[2]))[1]); sys.exit(0)
     print(json.dumps(analyse(sys.argv[1]), indent=2))

deploy.sh

@@ -46,7 +46,7 @@ apply_changes() {
   if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
 
   # 1. host scripts + units
-  if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py /usr/local/bin/; fi
+  if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi
   if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
   if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
   if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi

install.sh

@@ -48,7 +48,7 @@ for d in /srv/media/clips/*/; do
 done
 
 echo "== host scripts + units"
-install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py /usr/local/bin/
+install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
 install -d -m 0750 /etc/vanguard
 install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
 install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/