techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add vgbot-merge-sqlite.py: merge the bot's old SQLite history into the live MariaDB

Browse Source 
The bot's own importer requires an empty destination, but the bot rebuilds part of its state from Discord on
every start, so after the backend switch neither side is a superset. This merges them: log tables append on a
natural key, and the profile/state tables resolve column by column with one explicit rule per column, so the
original joined_at, first_seen_at, rank started_at and tracking start survive a rebuild that stamped today,
while current rank, presence and latest activity stay fresh.

One transaction, idempotent, and it stops rather than guessing when a column has no rule or a table is absent
from the destination. Tested against a fixture built from the real 1041-row file plus a simulated rebuild:
history restored, fresh values kept, 30 membership events (29 historical + tonight's), re-run a no-op, and
both guards verified. A KeyError on insert-only tables during the second run was found and fixed that way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-03 00:01:19 -06:00
parent be12a34f8c
commit 5c91fa7e99
3 changed files with 187 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -278,12 +278,22 @@ On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by
By hand, any time:
bin/vgbot-merge-sqlite.py --source /data/<old>.db # merge an old SQLite database into the live MariaDB (report; --apply writes)
./deployBotBundle.sh vanguard-bot-bundle.zip # install a release (the artifact as Gitea hands it over, zip or tar.gz)
./deployBotBundle.sh --latest # fetch the newest main build first (needs BOT_REPO_TOKEN in .env)
./deployBotBundle.sh --status # version, commit, package list, container state
./deployBotBundle.sh --rollback # back to the previous bundle
docker compose logs -f discord-bot # watch it
**Merging an old SQLite database in** (`bin/vgbot-merge-sqlite.py`): the bot rebuilds part of its state from
Discord on every start, so after a backend switch some tables hold fresh rows while the old file holds months of
history, and neither side is a superset. The bot's own importer refuses a non-empty destination by design, so this
merges instead: log tables append on a natural key, and the handful of profile/state tables are resolved column by
column with an explicit rule each (earliest `joined_at`, `first_seen_at` and rank `started_at` win, so a rebuild
that stamped today's date is undone; latest activity and current rank/presence win). It runs in one transaction,
is idempotent, and refuses rather than guessing if a column has no rule or a table is missing from the
destination. Run it with the gateway stopped, report first, then `--apply`.
The bot serves **both** its API and its web pages on 127.0.0.1:8000, published as `https://bot.HOST/` behind
**Authentik forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`, and the
bot authorises per request from those headers (`WEB_ALLOWED_GROUPS=bot-users,admins`; `WEB_SERVE_UI=true`