Plan: off-site gap filling option (home NVENC / Spot GPU) and SYS-3 availability note

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -357,6 +357,17 @@ Checklist if adopted:
 - [ ] per-user clips volume in `copyparty.conf` once forward-auth is live (Phase 5)
 
 ### Later — when it hurts
+
+- [ ] **Off-site gap filling** (instead of burning the SYS-1's CPU): streams whose sidecar reports
+      `gaps.total_missing_ms` ≥ threshold are re-encoded elsewhere with a hardware encoder.
+      - the sweep keeps the raw recording in `.src/` and the sidecar flags it (`needs_fill: true`)
+      - a nightly job on the home box (RTX 2080 Ti, NVENC) or a Spot GPU instance pulls flagged raws over
+        SFTP, re-encodes to CFR with duplicated frames (same editing standard, `tmcd` + `creation_time`
+        carried from the sidecar), pushes the result back over the published file, clears the flag
+      - Jellyfin keeps the same path → item identity and watched state survive; the plugin re-reads metadata
+      - until then: `FILL_GAPS_MS` stays off; players hold frames across gaps, editors see the gap list
+- [ ] **SYS-3 (E-2288G, Quick Sync)** — the 2×6 TB disk option is currently unavailable; re-check and swap when
+      it returns. Then: Jellyfin transcoding on, sweep re-encodes in minutes, `FILL_GAPS_MS` can simply be on.
 - [ ] **SYS-3 (E-2288G)**: Quick Sync → Jellyfin transcoding on, sweep re-encodes in minutes.
       Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
 - [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron

README.md

@@ -315,6 +315,24 @@ Viewing: the live portal crops the band; Jellyfin shows it as a thin strip at th
 Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
 Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
 
+Dropped frames / gaps: OBS keeps stamping frames from its clock, so after a drop the file has a hole and every
+frame that exists is still at the right time — the timecode stays correct, and players (Jellyfin, the portal)
+hold the last frame across the hole with audio in sync. By default the sweep does **not** re-encode a stream to
+fill holes (hours of CPU on this box); the sidecar lists them (`gaps.count`, `total_missing_ms`, first 50
+positions). Premiere preserves timing on such files; in Resolve check the clip's frame-rate interpretation.
+**Gap filling (available, off by default).** The sweep can re-encode a stream whose holes total at least
+`FILL_GAPS_MS` milliseconds to constant frame rate with duplicated frames (raw kept in `.src/`, timecode carried
+over). It costs ~2× the stream length in CPU on the E-2136, so it is off until the box has a hardware encoder.
+Turn it on (threshold 1 s) / off, on the server:
+
+    sudo sed -i 's|^#* *Environment=FILL_GAPS_MS=.*|Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
+    sudo sed -i 's|^Environment=FILL_GAPS_MS=.*|# Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
+
+Check: `systemctl show normalize-sweep -p Environment`. Takes effect on the next sweep run; already-published
+files are not revisited. Clips are always converted to CFR regardless.
+Planned alternative: re-encode flagged streams off-site on a machine with a hardware encoder (home box NVENC or a
+Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
+
 ## Layout on the server
 
     /srv/media/streams/<user>/<Title>/.live/<ts>.mp4             recording in progress (hidden from Jellyfin/copyparty)

STATUS.md

@@ -76,13 +76,14 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
 9. Clip submissions (belongs to the bot). Bot code from the other dev.
 10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
-11. SYS-3 when transcoding or sweep speed hurts.
+11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
 
 ## Verified behaviours worth remembering
 - **Never scp generated configs** (`mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt`, `.env`): a template copy wiped all streamers once. They are gitignored now; `*.example.*` are reference only.
 - Caddy runs in a container: anything it serves from disk must be a mount (`/srv/live` was missing → 404).
 - Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
 - Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
+- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
 - The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
 - A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
 - MediaMTX hot-reloads `paths`/recording settings but NOT `authMethod`/auth settings: those need a container restart (between streams).

bin/normalize-sweep.sh

@@ -19,6 +19,10 @@ set -uo pipefail
 
 ROOT="${MEDIA_ROOT:-/srv/media}"
 MINS_IDLE=3
+# Gap filling for STREAMS: re-encode to constant frame rate (duplicating frames across dropped-frame holes) when
+# the total missing time is >= this many ms. 0 = never (default; a 3 h re-encode is ~6 h of CPU on the E-2136).
+# Set via systemd Environment=FILL_GAPS_MS=... ; viewers already see held frames either way.
+FILL_GAPS_MS="${FILL_GAPS_MS:-0}"
 MAX_W=2560; MAX_H=1440; MAX_FPS=60
 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
 flock -n 9 || exit 0
@@ -61,7 +65,9 @@ probe() {
                med=d[int(n/2)]; if (med<=0) { print "cfr"; exit }
                bad=0; for(i=0;i<n;i++){ x=d[i]-med; if(x<0)x=-x; if (x > med*0.25) bad++ }
                print (bad > n*0.02) ? "vfr" : "cfr" }')
-  [[ $vfr == vfr ]] && { echo "encode vfr"; return; }
+  # streams: gaps from dropped frames are legitimate and re-encoding a 3 h file is hours of CPU — never VFR-encode
+  # them (the timecode sidecar lists the gaps instead). Clips (short, often genuinely VFR) still get CFR.
+  if [[ $vfr == vfr && $f != "$ROOT/streams/"* ]]; then echo "encode vfr"; return; fi
   local moov mdat; moov=$(grep -o '[0-9]*:moov' <<<"$head" | cut -d: -f1 | head -1); mdat=$(grep -o '[0-9]*:mdat' <<<"$head" | cut -d: -f1 | head -1)
   if [[ -z $moov || ( -n $mdat && $mdat -lt $moov ) ]]; then echo "remux index-last"; return; fi
   echo "ok"
@@ -126,6 +132,13 @@ PYEOF
       tcjson="${TC_JSON:-}"
       log "tc     $src  source=$TC_SOURCE timecode=$TC_TIMECODE"
     fi
+    # opt-in gap filling: switch this stream from remux to a CFR re-encode when the holes add up to enough
+    if (( FILL_GAPS_MS > 0 )) && [[ $verdict == remux && -n $tcjson ]]; then
+      missing=$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get("gaps",{}).get("total_missing_ms",0))' "$tcjson" 2>/dev/null || echo 0)
+      if (( missing >= FILL_GAPS_MS )); then
+        verdict=encode; reason="fill-gaps ${missing}ms"; log "gaps   $src  ${missing} ms missing -> CFR re-encode (FILL_GAPS_MS=$FILL_GAPS_MS)"
+      fi
+    fi
   fi
   if [[ $verdict == remux ]]; then
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
@@ -150,7 +163,13 @@ PYEOF
       if [[ -n $tcjson ]]; then                        # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
         printf '%s\n' "$tcjson" > "${out%.*}.json"; chown 1000:1000 "${out%.*}.json" 2>/dev/null
       fi
-      rm -f "$src"; rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
+      if [[ $verdict == encode ]]; then                # gap-filled re-encode: keep the raw recording in the hidden .src/
+        srcdir="$(dirname "$out")/.src"; mkdir -p "$srcdir"; chown 1000:1000 "$srcdir" 2>/dev/null
+        mv -f "$src" "$srcdir/$(basename "$out" .mp4).raw.mp4"
+      else
+        rm -f "$src"
+      fi
+      rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
       log "done   $out"; continue
     elif (( synced )); then
       rel_src="${src#"$stroot"/}"; rel_out="${out#"$stroot"/}"

bin/vgstrip.py

@@ -68,10 +68,15 @@ def decode_band_row(row: bytes, w: int):
 
 def probe(path):
     out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
-                          "stream=width,height,avg_frame_rate:format=duration", "-of", "json", path],
+                          "stream=width,height,avg_frame_rate,r_frame_rate:format=duration", "-of", "json", path],
                          capture_output=True, text=True, check=True).stdout
     j = json.loads(out); st = j["streams"][0]
-    num, den = st["avg_frame_rate"].split("/"); fps = float(num) / float(den or 1)
+    # nominal rate (r_frame_rate) — avg_frame_rate is skewed by dropped frames / gaps
+    def rate(x):
+        try: num, den = x.split("/"); return float(num) / float(den or 1)
+        except Exception: return 0.0
+    fps = rate(st.get("r_frame_rate", "")) or rate(st.get("avg_frame_rate", "")) or 60.0
+    if fps > 240: fps = rate(st.get("avg_frame_rate", "")) or 60.0      # bogus r_frame_rate on some files
     return int(st["width"]), int(st["height"]), fps, float(j["format"]["duration"])
 
 def sample_window(path, w, h, fps, start_s, dur_s):
@@ -120,15 +125,30 @@ def analyse(path, win_s=4.0):
     return {"source": "strip", "t0_ms": t0, "drift_ms": drift, "fps": fps, "width": w, "height": h,
             "duration_s": dur, "samples": samples}
 
+def find_gaps(path, fps, max_list=50):
+    """Holes in the video timeline (dropped frames): deltas > 1.5 frame periods. Demux only, fast."""
+    out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "packet=pts_time",
+                          "-of", "csv=p=0", path], capture_output=True, text=True).stdout
+    pts = sorted(float(x.strip(",").split(",")[0]) for x in out.split() if x.strip(",") not in ("", "N/A"))
+    period = 1.0 / (fps or 60.0)
+    gaps, total = [], 0.0
+    for a, b in zip(pts, pts[1:]):
+        d = b - a
+        if d > period * 1.5:
+            total += d - period
+            if len(gaps) < max_list: gaps.append({"at_s": round(a, 3), "missing_ms": int(round((d - period) * 1000))})
+    return {"count": sum(1 for a, b in zip(pts, pts[1:]) if b - a > period * 1.5), "total_missing_ms": int(round(total * 1000)), "first": gaps}
+
 def timecode_string(t0_ms, fps, tz="America/New_York"):
     """Time-of-day timecode for the tmcd track: HH:MM:SS:FF (drop-frame ';' for 29.97/59.94)."""
     from datetime import datetime, timezone
     from zoneinfo import ZoneInfo
     local = datetime.fromtimestamp(t0_ms / 1000.0, timezone.utc).astimezone(ZoneInfo(tz))
     frac = (t0_ms % 1000) / 1000.0
-    nominal = round(fps)
+    nominal = max(1, round(fps))
     ff = min(nominal - 1, int(frac * fps))
-    sep = ";" if abs(fps - nominal) > 0.01 else ":"
+    drop = any(abs(fps - r) < 0.05 for r in (29.97, 59.94, 23.976))   # true NTSC rates only
+    sep = ";" if drop else ":"
     return f"{local:%H:%M:%S}{sep}{ff:02d}", local
 
 def plan(path, arrival_ms=None, tz="America/New_York"):
@@ -142,14 +162,18 @@ def plan(path, arrival_ms=None, tz="America/New_York"):
     else:
         t0 = None; src = "none"
     out = {"source": src, "fps": r.get("fps"), "drift_ms": r.get("drift_ms", 0), "samples": r.get("samples", [])}
+    try:
+        out["gaps"] = find_gaps(path, r.get("fps") or 60.0)
+    except Exception:
+        pass
     if t0 is not None:
         tc, local = timecode_string(t0, r.get("fps") or 60.0, tz)
         utc = datetime.fromtimestamp(t0 / 1000.0, timezone.utc)
         out.update({"t0_ms": t0, "timestamp": utc.strftime("%Y-%m-%dT%H:%M:%S.") + f"{t0 % 1000:03d}Z",
                     "timecode": tc, "timecode_tz": tz, "local_start": local.isoformat()})
-        print(f"TC_TIMECODE={tc}")
-        print(f"TC_CREATION={utc.strftime('%Y-%m-%dT%H:%M:%S')}.{t0 % 1000:03d}Z")
-    print(f"TC_SOURCE={src}")
+        print(f"TC_TIMECODE='{tc}'")              # quoted: drop-frame uses ';' which the shell would split on
+        print(f"TC_CREATION='{utc.strftime('%Y-%m-%dT%H:%M:%S')}.{t0 % 1000:03d}Z'")
+    print(f"TC_SOURCE='{src}'")
     print("TC_JSON=" + json.dumps(json.dumps(out)))      # double-encoded: shell-safe single token
     return out
 

systemd/normalize-sweep.service

@@ -8,3 +8,4 @@ Type=oneshot
 ExecStart=/usr/local/bin/normalize-sweep.sh
 Nice=15
 IOSchedulingClass=idle
+# Environment=FILL_GAPS_MS=1000   # opt-in: CFR re-encode streams whose dropped-frame holes total >= 1 s (hours of CPU on this box)