Brandyn / Techy
b38d40ab0c
Observability was running but could not report anything. This closes that end to end and makes every piece of it reproducible from the installers instead of from a shell history. Logs - Filebeat 9 REMOVED the container input; it refuses the config per container and then runs with zero harvesters and no index. Container logs now use filestream + the container parser with a stable id each, and add_docker_metadata is told where the id is. - Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket. - Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever. Security - xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether the user holds a privilege and that API does not exist when security is off (the Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays the only gate. configure.py generates the service passwords, creates the accounts and writes a root-only netrc for the deadman; --es-security retries just that step. Alerting - A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers them to Discord. Alerts are therefore searchable beside the logs that caused them. - Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved. - The deadman watches both halves of that pipeline, since nothing inside the stack can report that the stack's own alerting died. Our own metrics - bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the measured answer to whether normalising keeps up, which was previously only a benchmark guess. - The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable. - Its file loop reads a process substitution, not a pipe, or the subshell discards every counter. Housekeeping - configure.py --apply installs the media scripts (it never did, so applies left stale binaries), creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup. - The discord-bot container asks the bot about itself via /health rather than probing a port. - Removed the dead diag landing page: Kibana occupies the root of that host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
55 lines
3.5 KiB
Bash
Executable File
55 lines
3.5 KiB
Bash
Executable File
#!/bin/bash
|
|
# One-off / catch-up: remove the timecode overscan band from stream recordings that were published BEFORE the
|
|
# sweep learned to crop it. Re-encodes (libx264, frame timing preserved, audio copied), re-applies the timecode
|
|
# from the sidecar, replaces the file, marks the sidecar band_removed. Skips files whose sidecar already says so.
|
|
# vgdeband.sh all of /srv/media/streams/*/*.mp4 with a strip sidecar
|
|
# vgdeband.sh FILE.mp4 ... just these
|
|
# Long: a 3 h 1440p60 night is a few hours of CPU. Run detached: sudo systemd-run --unit vgdeband -p Nice=15 \
|
|
# -p IOSchedulingClass=idle /usr/local/bin/vgdeband.sh ; follow with journalctl -fu vgdeband
|
|
set -uo pipefail
|
|
ROOT="${MEDIA_ROOT:-/srv/media}"
|
|
vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgdeband %s\n' "$(date -Is)" "$lvl" "$*"; }
|
|
log() { vglog info "msg=\"$*\""; }
|
|
dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
|
|
# Metrics (LOGGING.md): the same counters the sweep keeps, under component=vgdeband. No-ops if not installed.
|
|
if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
|
|
VG_MET_COMPONENT=vgdeband
|
|
. "$(dirname "$0")/vgmetrics.sh" && vgmet_init
|
|
else
|
|
vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
|
|
fi
|
|
if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi
|
|
for f in "${files[@]}"; do
|
|
side="${f%.*}.json"
|
|
[[ -f $side ]] || { vglog info "event=skip reason=no_sidecar file=\"$f\""; continue; }
|
|
read -r src removed tc ctime < <(python3 -c '
|
|
import json,sys; d=json.load(open(sys.argv[1]))
|
|
print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode",""), d.get("timestamp",""))' "$side")
|
|
[[ $src == strip ]] || { vglog info "event=skip reason=source_$src file=\"$f\""; continue; }
|
|
(( removed )) && { vglog info "event=skip reason=already_removed file=\"$f\""; continue; }
|
|
fuser -s "$f" 2>/dev/null && { vglog info "event=skip reason=in_use file=\"$f\""; continue; }
|
|
w=$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=p=0 "$f")
|
|
band=$(python3 /usr/local/bin/vgstrip.py band "$w")
|
|
tmp="$f.part"
|
|
tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime")
|
|
vglog info "event=deband file=\"$f\" width=$w band_rows=$band timecode=\"$tc\""
|
|
job_t0=$(vgmet_time)
|
|
nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \
|
|
-map 0:v:0 -map '0:a?' \
|
|
-c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "${ENCODE_THREADS:-2}" \
|
|
-vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \
|
|
"${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
|
|
rc=$?; job_secs=$(vgmet_since "$job_t0"); d1=$(dur "$f"); d2=$(dur "$tmp")
|
|
vgmet_add "seconds:encode:stream" "$job_secs"
|
|
if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
|
|
vgmet_add "jobs:encode:done:stream"; vgmet_add deband 1
|
|
mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null
|
|
python3 -c '
|
|
import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True
|
|
json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null
|
|
vglog info "event=done file=\"$f\""
|
|
else
|
|
rm -f "$tmp"; vgmet_add "jobs:encode:failed:stream"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2
|
|
fi
|
|
done
|