# Local /tldr benchmark

Runs the bot's real summarizing pipeline (trimming, tags, prompt, streaming)
against one or more Ollama models and writes a self-contained HTML dashboard.
Everything stays on your machine: the chat is read from disk, the only network
traffic is to your Ollama server, and the dashboard makes no requests at all.

## Run it

Open the tunnel to the ThinkCentre (keep it running in another terminal):

```bash
ssh -N -L 11500:localhost:11434 user@host
```

Then, from `velox-bot/`:

```bash
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.txt \
TLDR_BENCH_MODELS=qwen3:4b-instruct-2507-q4_K_M \
go test -tags bench ./internal/tldr -run TestBench -v -timeout 2h -count=1
```

It prints the dashboard path when it finishes (and keeps it updated while it
runs). Open the `.html` file in a browser. A `.json` with the raw results is
written next to it.

Try it first with the bundled example: `TLDR_BENCH_CHAT=internal/tldr/tldrbench/testdata/example-chat.txt`
and `TLDR_BENCH_FACTS=internal/tldr/tldrbench/testdata/example-facts.json`.

## Settings (environment variables)

Relative paths are relative to `velox-bot/`.

| Variable | Default | Meaning |
|---|---|---|
| `TLDR_BENCH_CHAT` | required | The chat dump (formats below). |
| `TLDR_BENCH_MODELS` | required | Comma separated Ollama model names. |
| `TLDR_BENCH_URL` | `http://localhost:11500` | Ollama address (the tunnel). |
| `TLDR_BENCH_RUNS` | `2` | Runs per model and prompt. |
| `TLDR_BENCH_THREADS` | `8` | CPU threads Ollama may use. |
| `TLDR_BENCH_FACTS` | none | Facts file, see below. Without it you only get timings and the automatic checks. |
| `TLDR_BENCH_PROMPTS` | none | A folder of `.txt` files, each a replacement system prompt to compare against `current`. A line `---USER---` splits a file into the system prompt and a reminder appended to every user message. |
| `TLDR_BENCH_SKIP_CURRENT` | off | Set to `1` to skip the bot's own prompt and run only the ones in `TLDR_BENCH_PROMPTS`. |
| `TLDR_BENCH_OUT` | `bench-out` | Where results are written. Also gets `current-prompt.txt` to copy from. |
| `TLDR_BENCH_OMIT_TEXT` | off | Set to `1` to leave the model outputs out of the dashboard and JSON, e.g. to share them. |

## Chat formats

- **Text copied from Discord**: select messages in the client and paste them
  into a file. `Name — Today at 14:32` header lines are recognised.
- **Plain lines**: `Author: message`. A line without an author continues the
  previous message.
- **DiscordChatExporter JSON** (`.json`): keeps replies, attachments and link
  preview titles, so it tests the tags too.

Put private dumps in `internal/tldr/tldrbench/private/` or `bench-out/`; both
are git-ignored.

## Facts file

A JSON list. A fact passes when one bullet of the overview matches every
pattern (case-insensitive regular expressions, `a|b` for alternatives):

```json
[
  {"name": "Dinner Friday 20h", "all": ["jantar", "sexta", "20h"]},
  {"name": "Hosting 80 to 120", "all": ["120", "80"]}
]
```

Generate a starting point first:

```bash
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.json \
go test -tags bench ./internal/tldr -run TestFactsSkeleton -v -count=1
```

It prints the chat lines that carry figures, times, days, money or links, with
suggested patterns (a topic word plus the figures and days in the line), and
writes them to `bench-out/facts-skeleton.json`. It does not understand the
chat, so it over-suggests: copy the file, delete the weak candidates, merge
lines about the same topic into one fact, and rename them. `TLDR_BENCH_CANDIDATES`
(default 40) limits how many it proposes.

Write 10 to 20 facts from reading the chat yourself: the decisions, plans with a
date or place, and figures. Requiring several details in one bullet is what
catches summaries that mix topics.

## What the dashboard checks without a facts file

- **Numbers not in the chat**: figures in the overview that appear nowhere in
  the transcript. A cheap detector for invented details.
- **Copies the chat**: one huge bullet, or a pasted transcript.
- **English intro**: starts with "Here is an overview...".
- **May be cut off**: ends without punctuation, usually the token limit.
- **Seconds** to finish and to the first streamed text.

Treat small gaps with care: two runs per cell at temperature 0.3 is noisy, so
raise `TLDR_BENCH_RUNS` to 3 or 4 for decisions.