FernandoJVideiraandClaude Opus 5.5
17548bbf7d
`go test -tags bench ./internal/tldr -run TestBench` runs the real /tldr pipeline against one or more Ollama models and prompt variants on a chat dump from disk (text copied from Discord, "Author: message" lines, or a DiscordChatExporter JSON export) and writes a self-contained HTML dashboard plus a JSON of the results. Scoring uses an optional facts file (a fact passes only when one bullet holds all its details) and automatic checks: numbers not in the chat, copying the chat, an English intro, cut-off output. TestFactsSkeleton proposes candidate facts from the chat to prune by hand. Chat dumps and results stay local: bench-out/ and the private folders are git-ignored. See internal/tldr/tldrbench/README.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
101 lines
4.3 KiB
Markdown
101 lines
4.3 KiB
Markdown
# Local /tldr benchmark
|
|
|
|
Runs the bot's real summarizing pipeline (trimming, tags, prompt, streaming)
|
|
against one or more Ollama models and writes a self-contained HTML dashboard.
|
|
Everything stays on your machine: the chat is read from disk, the only network
|
|
traffic is to your Ollama server, and the dashboard makes no requests at all.
|
|
|
|
## Run it
|
|
|
|
Open the tunnel to the ThinkCentre (keep it running in another terminal):
|
|
|
|
```bash
|
|
ssh -N -L 11500:localhost:11434 user@host
|
|
```
|
|
|
|
Then, from `velox-bot/`:
|
|
|
|
```bash
|
|
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.txt \
|
|
TLDR_BENCH_MODELS=qwen3:4b-instruct-2507-q4_K_M \
|
|
go test -tags bench ./internal/tldr -run TestBench -v -timeout 2h -count=1
|
|
```
|
|
|
|
It prints the dashboard path when it finishes (and keeps it updated while it
|
|
runs). Open the `.html` file in a browser. A `.json` with the raw results is
|
|
written next to it.
|
|
|
|
Try it first with the bundled example: `TLDR_BENCH_CHAT=internal/tldr/tldrbench/testdata/example-chat.txt`
|
|
and `TLDR_BENCH_FACTS=internal/tldr/tldrbench/testdata/example-facts.json`.
|
|
|
|
## Settings (environment variables)
|
|
|
|
Relative paths are relative to `velox-bot/`.
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `TLDR_BENCH_CHAT` | required | The chat dump (formats below). |
|
|
| `TLDR_BENCH_MODELS` | required | Comma separated Ollama model names. |
|
|
| `TLDR_BENCH_URL` | `http://localhost:11500` | Ollama address (the tunnel). |
|
|
| `TLDR_BENCH_RUNS` | `2` | Runs per model and prompt. |
|
|
| `TLDR_BENCH_THREADS` | `8` | CPU threads Ollama may use. |
|
|
| `TLDR_BENCH_FACTS` | none | Facts file, see below. Without it you only get timings and the automatic checks. |
|
|
| `TLDR_BENCH_PROMPTS` | none | A folder of `.txt` files, each a replacement system prompt to compare against `current`. A line `---USER---` splits a file into the system prompt and a reminder appended to every user message. |
|
|
| `TLDR_BENCH_SKIP_CURRENT` | off | Set to `1` to skip the bot's own prompt and run only the ones in `TLDR_BENCH_PROMPTS`. |
|
|
| `TLDR_BENCH_OUT` | `bench-out` | Where results are written. Also gets `current-prompt.txt` to copy from. |
|
|
| `TLDR_BENCH_OMIT_TEXT` | off | Set to `1` to leave the model outputs out of the dashboard and JSON, e.g. to share them. |
|
|
|
|
## Chat formats
|
|
|
|
- **Text copied from Discord**: select messages in the client and paste them
|
|
into a file. `Name — Today at 14:32` header lines are recognised.
|
|
- **Plain lines**: `Author: message`. A line without an author continues the
|
|
previous message.
|
|
- **DiscordChatExporter JSON** (`.json`): keeps replies, attachments and link
|
|
preview titles, so it tests the tags too.
|
|
|
|
Put private dumps in `internal/tldr/tldrbench/private/` or `bench-out/`; both
|
|
are git-ignored.
|
|
|
|
## Facts file
|
|
|
|
A JSON list. A fact passes when one bullet of the overview matches every
|
|
pattern (case-insensitive regular expressions, `a|b` for alternatives):
|
|
|
|
```json
|
|
[
|
|
{"name": "Dinner Friday 20h", "all": ["jantar", "sexta", "20h"]},
|
|
{"name": "Hosting 80 to 120", "all": ["120", "80"]}
|
|
]
|
|
```
|
|
|
|
Generate a starting point first:
|
|
|
|
```bash
|
|
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.json \
|
|
go test -tags bench ./internal/tldr -run TestFactsSkeleton -v -count=1
|
|
```
|
|
|
|
It prints the chat lines that carry figures, times, days, money or links, with
|
|
suggested patterns (a topic word plus the figures and days in the line), and
|
|
writes them to `bench-out/facts-skeleton.json`. It does not understand the
|
|
chat, so it over-suggests: copy the file, delete the weak candidates, merge
|
|
lines about the same topic into one fact, and rename them. `TLDR_BENCH_CANDIDATES`
|
|
(default 40) limits how many it proposes.
|
|
|
|
Write 10 to 20 facts from reading the chat yourself: the decisions, plans with a
|
|
date or place, and figures. Requiring several details in one bullet is what
|
|
catches summaries that mix topics.
|
|
|
|
## What the dashboard checks without a facts file
|
|
|
|
- **Numbers not in the chat**: figures in the overview that appear nowhere in
|
|
the transcript. A cheap detector for invented details.
|
|
- **Copies the chat**: one huge bullet, or a pasted transcript.
|
|
- **English intro**: starts with "Here is an overview...".
|
|
- **May be cut off**: ends without punctuation, usually the token limit.
|
|
- **Seconds** to finish and to the first streamed text.
|
|
|
|
Treat small gaps with care: two runs per cell at temperature 0.3 is noisy, so
|
|
raise `TLDR_BENCH_RUNS` to 3 or 4 for decisions.
|