velox-bot/internal/tldr/tldrbench/README.md

Local /tldr benchmark

Runs the bot's real summarizing pipeline (trimming, tags, prompt, streaming) against one or more Ollama models and writes a self-contained HTML dashboard. Everything stays on your machine: the chat is read from disk, the only network traffic is to your Ollama server, and the dashboard makes no requests at all.

Run it

Open the tunnel to the ThinkCentre (keep it running in another terminal):

ssh -N -L 11500:localhost:11434 user@host

Then, from velox-bot/:

TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.txt \
TLDR_BENCH_MODELS=qwen3:4b-instruct-2507-q4_K_M \
go test -tags bench ./internal/tldr -run TestBench -v -timeout 2h -count=1

It prints the dashboard path when it finishes (and keeps it updated while it runs). Open the .html file in a browser. A .json with the raw results is written next to it.

Try it first with the bundled example: TLDR_BENCH_CHAT=internal/tldr/tldrbench/testdata/example-chat.txt and TLDR_BENCH_FACTS=internal/tldr/tldrbench/testdata/example-facts.json.

Settings (environment variables)

Relative paths are relative to velox-bot/.

Variable	Default	Meaning
TLDR_BENCH_CHAT	required	The chat dump (formats below).
TLDR_BENCH_MODELS	required	Comma separated Ollama model names.
TLDR_BENCH_URL	http://localhost:11500	Ollama address (the tunnel).
TLDR_BENCH_RUNS	2	Runs per model and prompt.
TLDR_BENCH_THREADS	8	CPU threads Ollama may use.
TLDR_BENCH_FACTS	none	Facts file, see below. Without it you only get timings and the automatic checks.
TLDR_BENCH_PROMPTS	none	A folder of .txt files, each a replacement system prompt to compare against current. A line ---USER--- splits a file into the system prompt and a reminder appended to every user message.
TLDR_BENCH_SKIP_CURRENT	off	Set to 1 to skip the bot's own prompt and run only the ones in TLDR_BENCH_PROMPTS.
TLDR_BENCH_OUT	bench-out	Where results are written. Also gets current-prompt.txt to copy from.
TLDR_BENCH_OMIT_TEXT	off	Set to 1 to leave the model outputs out of the dashboard and JSON, e.g. to share them.

Chat formats

• Text copied from Discord: select messages in the client and paste them into a file. Name — Today at 14:32 header lines are recognised.
• Plain lines: Author: message. A line without an author continues the previous message.
• DiscordChatExporter JSON (.json): keeps replies, attachments and link preview titles, so it tests the tags too.

Put private dumps in internal/tldr/tldrbench/private/ or bench-out/; both are git-ignored.

Facts file

A JSON list. A fact passes when one bullet of the overview matches every pattern (case-insensitive regular expressions, a|b for alternatives):

[
  {"name": "Dinner Friday 20h", "all": ["jantar", "sexta", "20h"]},
  {"name": "Hosting 80 to 120", "all": ["120", "80"]}
]

Generate a starting point first:

TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.json \
go test -tags bench ./internal/tldr -run TestFactsSkeleton -v -count=1

It prints the chat lines that carry figures, times, days, money or links, with suggested patterns (a topic word plus the figures and days in the line), and writes them to bench-out/facts-skeleton.json. It does not understand the chat, so it over-suggests: copy the file, delete the weak candidates, merge lines about the same topic into one fact, and rename them. TLDR_BENCH_CANDIDATES (default 40) limits how many it proposes.

Write 10 to 20 facts from reading the chat yourself: the decisions, plans with a date or place, and figures. Requiring several details in one bullet is what catches summaries that mix topics.

What the dashboard checks without a facts file

• Numbers not in the chat: figures in the overview that appear nowhere in the transcript. A cheap detector for invented details.
• Copies the chat: one huge bullet, or a pasted transcript.
• English intro: starts with "Here is an overview...".
• May be cut off: ends without punctuation, usually the token limit.
• Seconds to finish and to the first streamed text.

Treat small gaps with care: two runs per cell at temperature 0.3 is noisy, so raise TLDR_BENCH_RUNS to 3 or 4 for decisions.