Add a local benchmark for /tldr with a dashboard
`go test -tags bench ./internal/tldr -run TestBench` runs the real /tldr pipeline against one or more Ollama models and prompt variants on a chat dump from disk (text copied from Discord, "Author: message" lines, or a DiscordChatExporter JSON export) and writes a self-contained HTML dashboard plus a JSON of the results. Scoring uses an optional facts file (a fact passes only when one bullet holds all its details) and automatic checks: numbers not in the chat, copying the chat, an English intro, cut-off output. TestFactsSkeleton proposes candidate facts from the chat to prune by hand. Chat dumps and results stay local: bench-out/ and the private folders are git-ignored. See internal/tldr/tldrbench/README.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
FernandoJVideiraandClaude Opus 5.5 committed
1 parent
7ae1fa5747
commit
17548bbf7d
10 files changed
+1495
No files matched your search
@@ -0,0 +1,100 @@
|
||||
# Local /tldr benchmark
|
||||
|
||||
Runs the bot's real summarizing pipeline (trimming, tags, prompt, streaming)
|
||||
against one or more Ollama models and writes a self-contained HTML dashboard.
|
||||
Everything stays on your machine: the chat is read from disk, the only network
|
||||
traffic is to your Ollama server, and the dashboard makes no requests at all.
|
||||
|
||||
## Run it
|
||||
|
||||
Open the tunnel to the ThinkCentre (keep it running in another terminal):
|
||||
|
||||
```bash
|
||||
ssh -N -L 11500:localhost:11434 user@host
|
||||
```
|
||||
|
||||
Then, from `velox-bot/`:
|
||||
|
||||
```bash
|
||||
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.txt \
|
||||
TLDR_BENCH_MODELS=qwen3:4b-instruct-2507-q4_K_M \
|
||||
go test -tags bench ./internal/tldr -run TestBench -v -timeout 2h -count=1
|
||||
```
|
||||
|
||||
It prints the dashboard path when it finishes (and keeps it updated while it
|
||||
runs). Open the `.html` file in a browser. A `.json` with the raw results is
|
||||
written next to it.
|
||||
|
||||
Try it first with the bundled example: `TLDR_BENCH_CHAT=internal/tldr/tldrbench/testdata/example-chat.txt`
|
||||
and `TLDR_BENCH_FACTS=internal/tldr/tldrbench/testdata/example-facts.json`.
|
||||
|
||||
## Settings (environment variables)
|
||||
|
||||
Relative paths are relative to `velox-bot/`.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `TLDR_BENCH_CHAT` | required | The chat dump (formats below). |
|
||||
| `TLDR_BENCH_MODELS` | required | Comma separated Ollama model names. |
|
||||
| `TLDR_BENCH_URL` | `http://localhost:11500` | Ollama address (the tunnel). |
|
||||
| `TLDR_BENCH_RUNS` | `2` | Runs per model and prompt. |
|
||||
| `TLDR_BENCH_THREADS` | `8` | CPU threads Ollama may use. |
|
||||
| `TLDR_BENCH_FACTS` | none | Facts file, see below. Without it you only get timings and the automatic checks. |
|
||||
| `TLDR_BENCH_PROMPTS` | none | A folder of `.txt` files, each a replacement system prompt to compare against `current`. A line `---USER---` splits a file into the system prompt and a reminder appended to every user message. |
|
||||
| `TLDR_BENCH_SKIP_CURRENT` | off | Set to `1` to skip the bot's own prompt and run only the ones in `TLDR_BENCH_PROMPTS`. |
|
||||
| `TLDR_BENCH_OUT` | `bench-out` | Where results are written. Also gets `current-prompt.txt` to copy from. |
|
||||
| `TLDR_BENCH_OMIT_TEXT` | off | Set to `1` to leave the model outputs out of the dashboard and JSON, e.g. to share them. |
|
||||
|
||||
## Chat formats
|
||||
|
||||
- **Text copied from Discord**: select messages in the client and paste them
|
||||
into a file. `Name — Today at 14:32` header lines are recognised.
|
||||
- **Plain lines**: `Author: message`. A line without an author continues the
|
||||
previous message.
|
||||
- **DiscordChatExporter JSON** (`.json`): keeps replies, attachments and link
|
||||
preview titles, so it tests the tags too.
|
||||
|
||||
Put private dumps in `internal/tldr/tldrbench/private/` or `bench-out/`; both
|
||||
are git-ignored.
|
||||
|
||||
## Facts file
|
||||
|
||||
A JSON list. A fact passes when one bullet of the overview matches every
|
||||
pattern (case-insensitive regular expressions, `a|b` for alternatives):
|
||||
|
||||
```json
|
||||
[
|
||||
{"name": "Dinner Friday 20h", "all": ["jantar", "sexta", "20h"]},
|
||||
{"name": "Hosting 80 to 120", "all": ["120", "80"]}
|
||||
]
|
||||
```
|
||||
|
||||
Generate a starting point first:
|
||||
|
||||
```bash
|
||||
TLDR_BENCH_CHAT=internal/tldr/tldrbench/private/chat.json \
|
||||
go test -tags bench ./internal/tldr -run TestFactsSkeleton -v -count=1
|
||||
```
|
||||
|
||||
It prints the chat lines that carry figures, times, days, money or links, with
|
||||
suggested patterns (a topic word plus the figures and days in the line), and
|
||||
writes them to `bench-out/facts-skeleton.json`. It does not understand the
|
||||
chat, so it over-suggests: copy the file, delete the weak candidates, merge
|
||||
lines about the same topic into one fact, and rename them. `TLDR_BENCH_CANDIDATES`
|
||||
(default 40) limits how many it proposes.
|
||||
|
||||
Write 10 to 20 facts from reading the chat yourself: the decisions, plans with a
|
||||
date or place, and figures. Requiring several details in one bullet is what
|
||||
catches summaries that mix topics.
|
||||
|
||||
## What the dashboard checks without a facts file
|
||||
|
||||
- **Numbers not in the chat**: figures in the overview that appear nowhere in
|
||||
the transcript. A cheap detector for invented details.
|
||||
- **Copies the chat**: one huge bullet, or a pasted transcript.
|
||||
- **English intro**: starts with "Here is an overview...".
|
||||
- **May be cut off**: ends without punctuation, usually the token limit.
|
||||
- **Seconds** to finish and to the first streamed text.
|
||||
|
||||
Treat small gaps with care: two runs per cell at temperature 0.3 is noisy, so
|
||||
raise `TLDR_BENCH_RUNS` to 3 or 4 for decisions.
|
||||
@@ -0,0 +1,205 @@
|
||||
// Package tldrbench is the toolkit behind the local /tldr benchmark: it loads
|
||||
// a chat dump from disk, scores a model's overview and renders a dashboard.
|
||||
// None of it is used by the bot itself, and nothing here talks to the network,
|
||||
// so a private chat dump never leaves the machine running the benchmark.
|
||||
package tldrbench
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/bwmarrin/discordgo"
|
||||
)
|
||||
|
||||
// LoadChat reads a chat dump and returns its messages oldest first. Three
|
||||
// formats are recognised by content:
|
||||
//
|
||||
// - a DiscordChatExporter JSON export (a file whose root has "messages"),
|
||||
// - text copied out of the Discord client ("Name — Today at 14:32" header
|
||||
// lines followed by the message lines),
|
||||
// - plain "Author: message" lines, where a line that does not start with an
|
||||
// author is the continuation of the previous message.
|
||||
func LoadChat(path string) ([]*discordgo.Message, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var msgs []*discordgo.Message
|
||||
if strings.EqualFold(filepath.Ext(path), ".json") || strings.HasPrefix(strings.TrimSpace(string(raw)), "{") {
|
||||
msgs, err = parseExport(raw)
|
||||
} else {
|
||||
msgs = parseText(string(raw))
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
if len(msgs) == 0 {
|
||||
return nil, fmt.Errorf("%s: no messages found", path)
|
||||
}
|
||||
return msgs, nil
|
||||
}
|
||||
|
||||
type exportFile struct {
|
||||
Messages []struct {
|
||||
ID string `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Timestamp string `json:"timestamp"`
|
||||
Content string `json:"content"`
|
||||
Author struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Nickname string `json:"nickname"`
|
||||
IsBot bool `json:"isBot"`
|
||||
} `json:"author"`
|
||||
Attachments []struct {
|
||||
FileName string `json:"fileName"`
|
||||
} `json:"attachments"`
|
||||
Embeds []struct {
|
||||
Title string `json:"title"`
|
||||
} `json:"embeds"`
|
||||
Reference *struct {
|
||||
MessageID string `json:"messageId"`
|
||||
} `json:"reference"`
|
||||
} `json:"messages"`
|
||||
}
|
||||
|
||||
func parseExport(raw []byte) ([]*discordgo.Message, error) {
|
||||
var f exportFile
|
||||
if err := json.Unmarshal(raw, &f); err != nil {
|
||||
return nil, fmt.Errorf("not a chat export: %w", err)
|
||||
}
|
||||
byID := make(map[string]*discordgo.Message, len(f.Messages))
|
||||
msgs := make([]*discordgo.Message, 0, len(f.Messages))
|
||||
for i, e := range f.Messages {
|
||||
name := e.Author.Nickname
|
||||
if name == "" {
|
||||
name = e.Author.Name
|
||||
}
|
||||
id := e.ID
|
||||
if id == "" {
|
||||
id = strconv.Itoa(i + 1)
|
||||
}
|
||||
m := &discordgo.Message{
|
||||
ID: id,
|
||||
Content: e.Content,
|
||||
Author: &discordgo.User{ID: firstNonEmpty(e.Author.ID, name), Username: name, Bot: e.Author.IsBot},
|
||||
}
|
||||
if ts, err := time.Parse(time.RFC3339Nano, e.Timestamp); err == nil {
|
||||
m.Timestamp = ts
|
||||
}
|
||||
switch e.Type {
|
||||
case "", "Default":
|
||||
case "Reply":
|
||||
m.Type = discordgo.MessageTypeReply
|
||||
if e.Reference != nil {
|
||||
m.ReferencedMessage = byID[e.Reference.MessageID]
|
||||
}
|
||||
default:
|
||||
m.Type = discordgo.MessageTypeGuildMemberJoin // anything that is not a human message
|
||||
}
|
||||
for _, a := range e.Attachments {
|
||||
m.Attachments = append(m.Attachments, &discordgo.MessageAttachment{Filename: a.FileName})
|
||||
}
|
||||
for _, em := range e.Embeds {
|
||||
m.Embeds = append(m.Embeds, &discordgo.MessageEmbed{Type: discordgo.EmbedTypeArticle, Title: em.Title})
|
||||
}
|
||||
byID[id] = m
|
||||
msgs = append(msgs, m)
|
||||
}
|
||||
return msgs, nil
|
||||
}
|
||||
|
||||
func firstNonEmpty(a, b string) string {
|
||||
if a != "" {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
var (
|
||||
// "Name — Today at 14:32", "Name — 12/10/2026 14:32", "Name — 14:32".
|
||||
clientHeader = regexp.MustCompile(`^(.{1,40}?) [—–-] (?:Today at |Yesterday at |hoje às |ontem às )?(?:\d{1,4}[/.\-]\d{1,2}[/.\-]\d{1,4},? ?)?(?:at |às )?\d{1,2}:\d{2}(?:\s?[AP]M)?\s*$`)
|
||||
// Lone timestamps the client shows next to grouped messages: "[14:33]" or "14:33".
|
||||
loneTime = regexp.MustCompile(`^\[?\d{1,2}:\d{2}(?:\s?[AP]M)?\]?$`)
|
||||
authorLine = regexp.MustCompile(`^([^:\n]{1,32}): (.+)$`)
|
||||
)
|
||||
|
||||
func parseText(s string) []*discordgo.Message {
|
||||
lines := strings.Split(strings.ReplaceAll(s, "\r\n", "\n"), "\n")
|
||||
|
||||
headers := 0
|
||||
for _, ln := range lines {
|
||||
if clientHeader.MatchString(strings.TrimSpace(ln)) {
|
||||
headers++
|
||||
}
|
||||
}
|
||||
if headers >= 2 {
|
||||
return parseClientCopy(lines)
|
||||
}
|
||||
return parseAuthorLines(lines)
|
||||
}
|
||||
|
||||
type builder struct{ msgs []*discordgo.Message }
|
||||
|
||||
func (b *builder) add(author, text string) {
|
||||
text = strings.TrimSpace(text)
|
||||
if text == "" {
|
||||
return
|
||||
}
|
||||
id := strconv.Itoa(len(b.msgs) + 1)
|
||||
b.msgs = append(b.msgs, &discordgo.Message{
|
||||
ID: id,
|
||||
Content: text,
|
||||
Author: &discordgo.User{ID: author, Username: author},
|
||||
})
|
||||
}
|
||||
|
||||
func parseClientCopy(lines []string) []*discordgo.Message {
|
||||
var b builder
|
||||
author := ""
|
||||
for _, ln := range lines {
|
||||
t := strings.TrimSpace(ln)
|
||||
switch {
|
||||
case t == "" || loneTime.MatchString(t):
|
||||
case clientHeader.MatchString(t):
|
||||
author = clientHeader.FindStringSubmatch(t)[1]
|
||||
case author != "":
|
||||
b.add(author, t)
|
||||
}
|
||||
}
|
||||
return b.msgs
|
||||
}
|
||||
|
||||
func parseAuthorLines(lines []string) []*discordgo.Message {
|
||||
var b builder
|
||||
var author string
|
||||
var cur []string
|
||||
flush := func() {
|
||||
if author != "" {
|
||||
b.add(author, strings.Join(cur, "\n"))
|
||||
}
|
||||
author, cur = "", nil
|
||||
}
|
||||
for _, ln := range lines {
|
||||
t := strings.TrimSpace(ln)
|
||||
if t == "" {
|
||||
continue
|
||||
}
|
||||
if m := authorLine.FindStringSubmatch(t); m != nil {
|
||||
flush()
|
||||
author, cur = strings.TrimSpace(m[1]), []string{m[2]}
|
||||
continue
|
||||
}
|
||||
if author != "" {
|
||||
cur = append(cur, t)
|
||||
}
|
||||
}
|
||||
flush()
|
||||
return b.msgs
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package tldrbench
|
||||
|
||||
import "strconv"
|
||||
|
||||
func fmtFloat(f float64, prec int) string { return strconv.FormatFloat(f, 'f', prec, 64) }
|
||||
|
||||
const pageHTML = `<!doctype html>
|
||||
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>/tldr benchmark</title>
|
||||
<style>
|
||||
:root{--bg:#f4f6f8;--surface:#fff;--surface-2:#eceff3;--fg:#16202b;--muted:#5b6876;--line:#d9dfe6;--accent:#2f5fd0;--good:#1b7f4b;--good-bg:#e0f3e8;--warn:#a35a00;--warn-bg:#fbecd2;--bad:#b3261e;--bad-bg:#fbe2df;color-scheme:light}
|
||||
@media (prefers-color-scheme:dark){:root{--bg:#0f151c;--surface:#17202a;--surface-2:#1f2a36;--fg:#e6ecf2;--muted:#93a1b0;--line:#2a3745;--accent:#7fa2ff;--good:#5fd391;--good-bg:#153a28;--warn:#f0b45a;--warn-bg:#3d2d12;--bad:#ff8d84;--bad-bg:#44201d;color-scheme:dark}}
|
||||
*{box-sizing:border-box}
|
||||
body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.55 system-ui,-apple-system,"Segoe UI",sans-serif;padding:32px max(16px,env(safe-area-inset-right,0px)) 64px max(16px,env(safe-area-inset-left,0px))}
|
||||
.wrap{max-width:1100px;margin:0 auto;display:flex;flex-direction:column;gap:28px}
|
||||
h1{font-size:1.9rem;margin:0;letter-spacing:-.02em}h2{font-size:1.1rem;margin:0 0 6px}
|
||||
.eyebrow{font:500 .74rem ui-monospace,Menlo,monospace;letter-spacing:.08em;text-transform:uppercase;color:var(--muted)}
|
||||
.sub,.note{color:var(--muted)}.note{font-size:.85rem}
|
||||
.panel{background:var(--surface);border:1px solid var(--line);border-radius:14px;padding:18px 20px;min-width:0}
|
||||
.tablewrap{overflow-x:auto}
|
||||
table{border-collapse:collapse;width:100%;font-size:.92rem}
|
||||
th{text-align:left;font:500 .76rem ui-monospace,Menlo,monospace;letter-spacing:.05em;text-transform:uppercase;color:var(--muted);padding:8px 12px;border-bottom:1px solid var(--line);white-space:nowrap}
|
||||
td{padding:10px 12px;border-bottom:1px solid var(--line);vertical-align:top}
|
||||
tr:last-child td{border-bottom:0}
|
||||
td.c,th.c{text-align:center}
|
||||
tr.group td{background:var(--surface-2);font:.78rem ui-monospace,Menlo,monospace;color:var(--muted);padding-block:6px}
|
||||
.pill{display:inline-block;font:500 .78rem ui-monospace,Menlo,monospace;padding:2px 10px;border-radius:999px;white-space:nowrap}
|
||||
.pill.ok{background:var(--good-bg);color:var(--good)}.pill.mid{background:var(--warn-bg);color:var(--warn)}.pill.no{background:var(--bad-bg);color:var(--bad)}.pill.na{background:var(--surface-2);color:var(--muted)}
|
||||
.h2{background:var(--good-bg);color:var(--good)}.h1{background:var(--warn-bg);color:var(--warn)}.h0{background:var(--bad-bg);color:var(--bad)}
|
||||
.mono{font-family:ui-monospace,Menlo,monospace;font-size:.84rem}
|
||||
details{background:var(--surface);border:1px solid var(--line);border-radius:10px;padding:0 16px;margin-top:8px}
|
||||
summary{cursor:pointer;padding:12px 0;font-weight:500}
|
||||
summary .meta{color:var(--muted);font:400 .78rem ui-monospace,Menlo,monospace;margin-left:8px}
|
||||
pre{margin:0 0 14px;padding:12px 14px;background:var(--surface-2);border-radius:8px;overflow-x:auto;white-space:pre-wrap;word-break:break-word;font:.8rem/1.6 ui-monospace,Menlo,monospace}
|
||||
.best{border-left:3px solid var(--accent);padding-left:12px}
|
||||
</style></head><body><div class="wrap">
|
||||
<div><span class="eyebrow">/tldr benchmark · local · {{.When.Format "2006-01-02 15:04"}}</span>
|
||||
<h1>{{len .Models}} model{{if ne (len .Models) 1}}s{{end}}, {{len .Variants}} prompt{{if ne (len .Variants) 1}}s{{end}}</h1>
|
||||
<p class="sub">Chat: <span class="mono">{{.Chat.Source}}</span>, {{.Chat.Messages}} messages, {{.Chat.Kept}} kept after trimming, {{.Chat.InputChars}} characters in {{.Chat.Chunks}} chunk{{if ne .Chat.Chunks 1}}s{{end}}{{if .Chat.Truncated}} (the transcript was truncated by the size limit){{end}}. {{.Threads}} threads. This page is self-contained and makes no network requests.</p></div>
|
||||
|
||||
{{if .BestModel}}<div class="panel best"><span class="eyebrow">Best on this chat</span><p><strong>{{.BestModel}}</strong> with prompt <strong>{{.BestVar}}</strong> got the most facts, preferring the faster one on a tie.</p></div>{{end}}
|
||||
|
||||
<section><h2>Summary</h2><p class="sub">Per cell: facts found{{if .Facts}} (out of {{len .Facts}}){{else}} (no facts file, so only the checks below){{end}}, average seconds to finish, seconds to the first text, and flags.</p>
|
||||
<div class="panel"><div class="tablewrap"><table>
|
||||
<thead><tr><th>Prompt</th>{{range .Models}}<th class="c">{{.}}</th>{{end}}</tr></thead><tbody>
|
||||
{{range .Rows}}<tr><td class="mono">{{.Variant}}</td>
|
||||
{{range .Cells}}<td class="c">{{if eq .Runs 0}}<span class="note">not run</span>{{else if eq .Runs .Failed}}<span class="pill no">failed</span>{{else}}
|
||||
{{if .HasFacts}}<span class="pill {{.Class}}">{{f1 .Facts}} of {{.FactsTotal}}</span>{{else}}<span class="pill na">{{f1 .Invented}} invented</span>{{end}}<br>
|
||||
<span class="note">{{sec .Seconds}} s{{if gt .FirstText 0.0}} · first text {{sec .FirstText}} s{{end}}</span>{{if .Flags}}<br><span class="note">{{range $i, $f := .Flags}}{{if $i}}, {{end}}{{$f}}{{end}}</span>{{end}}{{end}}</td>{{end}}</tr>{{end}}
|
||||
</tbody></table></div></div></section>
|
||||
|
||||
{{if .FactGrid}}<section><h2>Which facts each prompt got</h2><p class="sub">Each cell is how many of that cell's runs got the fact right.</p>
|
||||
<div class="panel"><div class="tablewrap"><table>
|
||||
<thead><tr><th>Prompt</th>{{range .Facts}}<th class="c">{{.}}</th>{{end}}</tr></thead><tbody>
|
||||
{{range .FactGrid}}<tr class="group"><td colspan="{{inc (len $.Facts)}}">{{.Model}}</td></tr>
|
||||
{{range .Rows}}<tr><td class="mono">{{.Variant}}</td>{{$runs := .Runs}}{{range .Hits}}<td class="c mono {{hitClass . $runs}}">{{.}}</td>{{end}}</tr>{{end}}{{end}}
|
||||
</tbody></table></div></div></section>{{end}}
|
||||
|
||||
{{if .HasText}}<section><h2>Every run</h2><p class="sub">Outputs exactly as returned.</p>
|
||||
{{range .RunsSorted}}<details><summary>{{.Model}} · {{.Variant}} · run {{.N}}<span class="meta">{{if .Error}}failed{{else}}{{sec .Seconds}} s{{if .Missed}} · missed: {{range $i, $m := .Missed}}{{if $i}}, {{end}}{{$m}}{{end}}{{end}}{{if .Invented}} · numbers not in the chat: {{range $i, $m := .Invented}}{{if $i}}, {{end}}{{$m}}{{end}}{{end}}{{end}}</span></summary>
|
||||
<pre>{{if .Error}}{{.Error}}{{else}}{{.Output}}{{end}}</pre></details>{{end}}</section>{{end}}
|
||||
|
||||
<p class="note">"Numbers not in the chat" lists figures the overview wrote that appear nowhere in the transcript: a cheap check for invented details. "May be cut off" means the text ends without punctuation, usually the token limit. Facts pass only when one bullet holds all of the fact's details.</p>
|
||||
</div></body></html>`
|
||||
@@ -0,0 +1,261 @@
|
||||
package tldrbench
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"html/template"
|
||||
"io"
|
||||
"os"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ChatInfo describes the chat that was summarized, without containing any of it.
|
||||
type ChatInfo struct {
|
||||
Source string `json:"source"` // file name only
|
||||
Messages int `json:"messages"`
|
||||
Kept int `json:"kept"`
|
||||
Chunks int `json:"chunks"`
|
||||
InputChars int `json:"inputChars"`
|
||||
Truncated bool `json:"truncated"`
|
||||
}
|
||||
|
||||
// Run is one model writing one overview with one prompt.
|
||||
type Run struct {
|
||||
Model string `json:"model"`
|
||||
Variant string `json:"variant"`
|
||||
N int `json:"n"`
|
||||
Seconds float64 `json:"seconds"`
|
||||
FirstText float64 `json:"firstText"` // seconds until the first streamed text, 0 if never
|
||||
Output string `json:"output,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Score
|
||||
}
|
||||
|
||||
// Report is everything the dashboard shows.
|
||||
type Report struct {
|
||||
When time.Time `json:"when"`
|
||||
Chat ChatInfo `json:"chat"`
|
||||
Threads int `json:"threads"`
|
||||
Facts []string `json:"facts,omitempty"`
|
||||
Models []string `json:"models"`
|
||||
Variants []string `json:"variants"`
|
||||
Runs []Run `json:"runs"`
|
||||
}
|
||||
|
||||
// WriteJSON saves the raw results so they can be re-rendered or compared later.
|
||||
func (r *Report) WriteJSON(path string) error {
|
||||
raw, err := json.MarshalIndent(r, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(path, raw, 0o600)
|
||||
}
|
||||
|
||||
// ReadReport loads a report written by WriteJSON.
|
||||
func ReadReport(path string) (*Report, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var r Report
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &r, nil
|
||||
}
|
||||
|
||||
// Cell aggregates the runs of one model with one prompt.
|
||||
type Cell struct {
|
||||
Runs, Failed int
|
||||
Seconds, FirstText, Facts float64
|
||||
Invented float64
|
||||
Copied, English, CutOff int
|
||||
HitCount map[string]int
|
||||
HasFacts bool
|
||||
Class string // ok, mid, no, na
|
||||
SecondsMin, SecondsMax float64
|
||||
FactsTotal int
|
||||
FactsPct int
|
||||
Flags []string
|
||||
}
|
||||
|
||||
func aggregate(runs []Run) Cell {
|
||||
c := Cell{HitCount: map[string]int{}, SecondsMin: -1}
|
||||
var ok int
|
||||
for _, r := range runs {
|
||||
c.Runs++
|
||||
if r.Error != "" {
|
||||
c.Failed++
|
||||
continue
|
||||
}
|
||||
ok++
|
||||
c.Seconds += r.Seconds
|
||||
c.FirstText += r.FirstText
|
||||
c.Facts += float64(r.FactsHit)
|
||||
c.Invented += float64(len(r.Invented))
|
||||
c.FactsTotal = r.FactsTotal
|
||||
if r.Copied {
|
||||
c.Copied++
|
||||
}
|
||||
if r.EnglishIntro {
|
||||
c.English++
|
||||
}
|
||||
if r.CutOff {
|
||||
c.CutOff++
|
||||
}
|
||||
for _, h := range r.Hit {
|
||||
c.HitCount[h]++
|
||||
}
|
||||
if c.SecondsMin < 0 || r.Seconds < c.SecondsMin {
|
||||
c.SecondsMin = r.Seconds
|
||||
}
|
||||
if r.Seconds > c.SecondsMax {
|
||||
c.SecondsMax = r.Seconds
|
||||
}
|
||||
}
|
||||
if ok > 0 {
|
||||
n := float64(ok)
|
||||
c.Seconds /= n
|
||||
c.FirstText /= n
|
||||
c.Facts /= n
|
||||
c.Invented /= n
|
||||
}
|
||||
c.HasFacts = c.FactsTotal > 0
|
||||
c.Class = "na"
|
||||
if ok > 0 && c.HasFacts {
|
||||
c.FactsPct = int(100*c.Facts/float64(c.FactsTotal) + 0.5)
|
||||
switch {
|
||||
case c.FactsPct >= 80:
|
||||
c.Class = "ok"
|
||||
case c.FactsPct >= 55:
|
||||
c.Class = "mid"
|
||||
default:
|
||||
c.Class = "no"
|
||||
}
|
||||
}
|
||||
if c.Copied > 0 {
|
||||
c.Flags = append(c.Flags, "copies the chat")
|
||||
}
|
||||
if c.English > 0 {
|
||||
c.Flags = append(c.Flags, "English intro")
|
||||
}
|
||||
if c.CutOff > 0 {
|
||||
c.Flags = append(c.Flags, "may be cut off")
|
||||
}
|
||||
if c.Invented > 0 {
|
||||
c.Flags = append(c.Flags, "numbers not in the chat")
|
||||
}
|
||||
if c.Failed > 0 {
|
||||
c.Flags = append(c.Flags, "failed runs")
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
type viewRow struct {
|
||||
Variant string
|
||||
Cells []Cell
|
||||
}
|
||||
|
||||
type viewModelFacts struct {
|
||||
Model string
|
||||
Rows []struct {
|
||||
Variant string
|
||||
Hits []int
|
||||
Runs int
|
||||
}
|
||||
}
|
||||
|
||||
type view struct {
|
||||
Report
|
||||
Rows []viewRow
|
||||
FactGrid []viewModelFacts
|
||||
BestModel string
|
||||
BestVar string
|
||||
HasText bool
|
||||
RunsSorted []Run
|
||||
}
|
||||
|
||||
// WriteHTML renders a self-contained dashboard: inline styles, no scripts, no
|
||||
// external requests.
|
||||
func (r *Report) WriteHTML(w io.Writer) error {
|
||||
v := view{Report: *r}
|
||||
cell := map[[2]string][]Run{}
|
||||
for _, run := range r.Runs {
|
||||
k := [2]string{run.Model, run.Variant}
|
||||
cell[k] = append(cell[k], run)
|
||||
if run.Output != "" {
|
||||
v.HasText = true
|
||||
}
|
||||
}
|
||||
best := -1.0
|
||||
for _, variant := range r.Variants {
|
||||
row := viewRow{Variant: variant}
|
||||
for _, model := range r.Models {
|
||||
c := aggregate(cell[[2]string{model, variant}])
|
||||
row.Cells = append(row.Cells, c)
|
||||
if c.HasFacts && c.Runs > c.Failed {
|
||||
score := c.Facts - c.Seconds/1000 // ties go to the faster one
|
||||
if score > best {
|
||||
best, v.BestModel, v.BestVar = score, model, variant
|
||||
}
|
||||
}
|
||||
}
|
||||
v.Rows = append(v.Rows, row)
|
||||
}
|
||||
if len(r.Facts) > 0 {
|
||||
for _, model := range r.Models {
|
||||
mf := viewModelFacts{Model: model}
|
||||
for _, variant := range r.Variants {
|
||||
c := aggregate(cell[[2]string{model, variant}])
|
||||
row := struct {
|
||||
Variant string
|
||||
Hits []int
|
||||
Runs int
|
||||
}{Variant: variant, Runs: c.Runs - c.Failed}
|
||||
for _, f := range r.Facts {
|
||||
row.Hits = append(row.Hits, c.HitCount[f])
|
||||
}
|
||||
mf.Rows = append(mf.Rows, row)
|
||||
}
|
||||
v.FactGrid = append(v.FactGrid, mf)
|
||||
}
|
||||
}
|
||||
v.RunsSorted = append(v.RunsSorted, r.Runs...)
|
||||
sort.SliceStable(v.RunsSorted, func(i, j int) bool {
|
||||
a, b := v.RunsSorted[i], v.RunsSorted[j]
|
||||
if a.Model != b.Model {
|
||||
return indexOf(r.Models, a.Model) < indexOf(r.Models, b.Model)
|
||||
}
|
||||
if a.Variant != b.Variant {
|
||||
return indexOf(r.Variants, a.Variant) < indexOf(r.Variants, b.Variant)
|
||||
}
|
||||
return a.N < b.N
|
||||
})
|
||||
return page.Execute(w, v)
|
||||
}
|
||||
|
||||
func indexOf(list []string, s string) int {
|
||||
for i, x := range list {
|
||||
if x == s {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return len(list)
|
||||
}
|
||||
|
||||
var page = template.Must(template.New("page").Funcs(template.FuncMap{
|
||||
"inc": func(n int) int { return n + 1 },
|
||||
"f1": func(f float64) string { return fmtFloat(f, 1) },
|
||||
"sec": func(f float64) string { return fmtFloat(f, 0) },
|
||||
"hitClass": func(hits, runs int) string {
|
||||
switch {
|
||||
case runs == 0:
|
||||
return "h0"
|
||||
case hits == runs:
|
||||
return "h2"
|
||||
case hits > 0:
|
||||
return "h1"
|
||||
}
|
||||
return "h0"
|
||||
},
|
||||
}).Parse(pageHTML))
|
||||
@@ -0,0 +1,169 @@
|
||||
package tldrbench
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// Fact is one thing a good overview must contain. It passes when a single
|
||||
// bullet matches every pattern, so a bullet that mentions the date of one
|
||||
// event next to the name of another does not count. Patterns are
|
||||
// case-insensitive regular expressions; use "a|b" for alternatives.
|
||||
type Fact struct {
|
||||
Name string `json:"name"`
|
||||
All []string `json:"all"`
|
||||
// Note is free text for you (SuggestFacts puts the chat line it came from
|
||||
// here); it is not used for scoring.
|
||||
Note string `json:"note,omitempty"`
|
||||
|
||||
res []*regexp.Regexp
|
||||
}
|
||||
|
||||
// LoadFacts reads a JSON list of facts and compiles their patterns.
|
||||
func LoadFacts(path string) ([]Fact, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var facts []Fact
|
||||
if err := json.Unmarshal(raw, &facts); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
for i := range facts {
|
||||
if facts[i].Name == "" || len(facts[i].All) == 0 {
|
||||
return nil, fmt.Errorf("%s: fact %d needs a name and at least one pattern", path, i+1)
|
||||
}
|
||||
for _, p := range facts[i].All {
|
||||
re, err := regexp.Compile("(?i)" + p)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: fact %q: %w", path, facts[i].Name, err)
|
||||
}
|
||||
facts[i].res = append(facts[i].res, re)
|
||||
}
|
||||
}
|
||||
return facts, nil
|
||||
}
|
||||
|
||||
// Score is how one overview did. Only the fact counts need a facts file; the
|
||||
// rest is judged from the overview and the chat alone.
|
||||
type Score struct {
|
||||
FactsHit int `json:"factsHit"`
|
||||
FactsTotal int `json:"factsTotal"`
|
||||
Hit []string `json:"hit,omitempty"`
|
||||
Missed []string `json:"missed,omitempty"`
|
||||
Bullets int `json:"bullets"`
|
||||
LongestChars int `json:"longestChars"`
|
||||
// Invented lists numbers in the overview that appear nowhere in the chat.
|
||||
Invented []string `json:"invented,omitempty"`
|
||||
Copied bool `json:"copied"` // looks like the chat pasted back, not a summary
|
||||
EnglishIntro bool `json:"englishIntro"` // opens with "Here is an overview..."
|
||||
CutOff bool `json:"cutOff"` // ends mid-sentence, usually the token limit
|
||||
}
|
||||
|
||||
var (
|
||||
bulletStart = regexp.MustCompile(`^([-*•]|\d+[.)])\s`)
|
||||
numberRe = regexp.MustCompile(`\d+(?:[.,:]\d+)*`)
|
||||
introRe = regexp.MustCompile(`(?i)^(here( i|')?s|here is|here are|sure|certainly|of course|this is an overview)`)
|
||||
)
|
||||
|
||||
// Bullets groups an overview into its bullets; lines that do not start one
|
||||
// continue the previous bullet.
|
||||
func Bullets(out string) []string {
|
||||
var bs []string
|
||||
for _, ln := range strings.Split(out, "\n") {
|
||||
s := strings.TrimSpace(ln)
|
||||
if s == "" {
|
||||
continue
|
||||
}
|
||||
if bulletStart.MatchString(s) || len(bs) == 0 {
|
||||
bs = append(bs, s)
|
||||
} else {
|
||||
bs[len(bs)-1] += " " + s
|
||||
}
|
||||
}
|
||||
return bs
|
||||
}
|
||||
|
||||
// normNumber makes "02", "2" and "2,0" compare equal enough for "was this
|
||||
// number in the chat".
|
||||
func normNumber(n string) string {
|
||||
n = strings.ReplaceAll(n, ",", ".")
|
||||
n = strings.TrimLeft(n, "0")
|
||||
if n == "" || n[0] == '.' || n[0] == ':' {
|
||||
n = "0" + n
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func numbers(s string) map[string]bool {
|
||||
set := map[string]bool{}
|
||||
for _, n := range numberRe.FindAllString(s, -1) {
|
||||
set[normNumber(n)] = true
|
||||
// "14:30" also makes "14" and "30" legitimate on their own.
|
||||
for _, part := range regexp.MustCompile(`[.,:]`).Split(n, -1) {
|
||||
set[normNumber(part)] = true
|
||||
}
|
||||
}
|
||||
return set
|
||||
}
|
||||
|
||||
// Evaluate scores an overview. chat is the transcript the model was given.
|
||||
func Evaluate(out, chat string, facts []Fact) Score {
|
||||
bullets := Bullets(out)
|
||||
var sc Score
|
||||
sc.Bullets = len(bullets)
|
||||
for _, b := range bullets {
|
||||
if n := len([]rune(b)); n > sc.LongestChars {
|
||||
sc.LongestChars = n
|
||||
}
|
||||
}
|
||||
|
||||
sc.FactsTotal = len(facts)
|
||||
for _, f := range facts {
|
||||
ok := false
|
||||
for _, b := range bullets {
|
||||
all := true
|
||||
for _, re := range f.res {
|
||||
if !re.MatchString(b) {
|
||||
all = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if all {
|
||||
ok = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if ok {
|
||||
sc.FactsHit++
|
||||
sc.Hit = append(sc.Hit, f.Name)
|
||||
} else {
|
||||
sc.Missed = append(sc.Missed, f.Name)
|
||||
}
|
||||
}
|
||||
|
||||
known := numbers(chat)
|
||||
seen := map[string]bool{}
|
||||
for _, b := range bullets {
|
||||
b = bulletStart.ReplaceAllString(b, "")
|
||||
for _, n := range numberRe.FindAllString(b, -1) {
|
||||
if !known[normNumber(n)] && !seen[n] {
|
||||
seen[n] = true
|
||||
sc.Invented = append(sc.Invented, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
trimmed := strings.TrimSpace(out)
|
||||
sc.Copied = (sc.Bullets <= 2 && len([]rune(trimmed)) > 600) || sc.LongestChars > 700
|
||||
sc.EnglishIntro = introRe.MatchString(trimmed)
|
||||
if r := []rune(trimmed); len(r) > 0 {
|
||||
last := r[len(r)-1]
|
||||
sc.CutOff = unicode.IsLetter(last) || unicode.IsDigit(last) || last == ',' || last == ';' || last == ':'
|
||||
}
|
||||
return sc
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package tldrbench
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
var (
|
||||
// Things a good overview has to get right: figures, times, money, days.
|
||||
figureRe = regexp.MustCompile(`\d+(?:[.,:]\d+)*(?:h\d*|€|%|k|gb|mb|min)?`)
|
||||
weekdayRe = regexp.MustCompile(`(?i)\b(segunda|ter[çc]a|quarta|quinta|sexta|s[áa]bado|domingo|monday|tuesday|wednesday|thursday|friday|saturday|sunday|amanh[ãa]|tomorrow|hoje|today)\b`)
|
||||
moneyRe = regexp.MustCompile(`(?i)€|\$|\beur\b|\beuros?\b`)
|
||||
wordRe = regexp.MustCompile(`[\p{L}][\p{L}'-]{5,}`)
|
||||
|
||||
// Common long words that make poor topic keywords.
|
||||
stopwords = map[string]bool{
|
||||
"porque": true, "também": true, "tambem": true, "depois": true, "ontem": true, "ainda": true, "estava": true,
|
||||
"podemos": true, "devíamos": true, "devemos": true, "fazer": true, "mesmo": true, "sempre": true, "ficou": true,
|
||||
"because": true, "should": true, "would": true, "could": true, "really": true, "maybe": true, "already": true,
|
||||
}
|
||||
)
|
||||
|
||||
// signals counts how many different "this matters" markers a line has.
|
||||
func signals(line string) int {
|
||||
n := 0
|
||||
if figureRe.MatchString(line) {
|
||||
n++
|
||||
}
|
||||
if weekdayRe.MatchString(line) {
|
||||
n++
|
||||
}
|
||||
if moneyRe.MatchString(line) {
|
||||
n++
|
||||
}
|
||||
if strings.Contains(line, "[link") || strings.Contains(line, "(preview:") {
|
||||
n++
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// SuggestFacts proposes candidate facts from transcript lines ("Author: text").
|
||||
// It does not understand the chat: it picks the lines that carry figures,
|
||||
// times, days, money or links, and for each suggests patterns (the figures and
|
||||
// days in it, plus one topic word). You are meant to prune and merge them into
|
||||
// the 10 to 20 facts that matter, then check the patterns against a real
|
||||
// overview. At most max candidates are returned, chronologically.
|
||||
func SuggestFacts(lines []string, max int) []Fact {
|
||||
type scored struct {
|
||||
idx, signals int
|
||||
line string
|
||||
}
|
||||
var cands []scored
|
||||
for i, ln := range lines {
|
||||
ln = strings.TrimSpace(ln)
|
||||
if ln == "" {
|
||||
continue
|
||||
}
|
||||
if s := signals(ln); s > 0 {
|
||||
cands = append(cands, scored{i, s, ln})
|
||||
}
|
||||
}
|
||||
// Keep the strongest, then restore chat order.
|
||||
sort.SliceStable(cands, func(a, b int) bool { return cands[a].signals > cands[b].signals })
|
||||
if max > 0 && len(cands) > max {
|
||||
cands = cands[:max]
|
||||
}
|
||||
sort.Slice(cands, func(a, b int) bool { return cands[a].idx < cands[b].idx })
|
||||
|
||||
var facts []Fact
|
||||
for n, c := range cands {
|
||||
facts = append(facts, Fact{
|
||||
Name: fmt.Sprintf("fact %d: %s", n+1, excerpt(c.line, 48)),
|
||||
All: suggestPatterns(c.line),
|
||||
Note: c.line,
|
||||
})
|
||||
}
|
||||
return facts
|
||||
}
|
||||
|
||||
func suggestPatterns(line string) []string {
|
||||
// Drop the "Author: " prefix so names do not become patterns.
|
||||
text := line
|
||||
if i := strings.Index(line, ": "); i > 0 && i <= 32 {
|
||||
text = line[i+2:]
|
||||
}
|
||||
|
||||
seen := map[string]bool{}
|
||||
var pats []string
|
||||
add := func(p string) {
|
||||
key := strings.ToLower(p)
|
||||
if p == "" || len(pats) >= 3 {
|
||||
return
|
||||
}
|
||||
for have := range seen {
|
||||
if strings.Contains(have, key) || strings.Contains(key, have) {
|
||||
return // already covered, e.g. "segunda" next to "segunda-feira"
|
||||
}
|
||||
}
|
||||
seen[key] = true
|
||||
pats = append(pats, p)
|
||||
}
|
||||
|
||||
// A topic word first, so the fact stays tied to its subject.
|
||||
best := ""
|
||||
for _, w := range wordRe.FindAllString(text, -1) {
|
||||
lw := strings.ToLower(w)
|
||||
if stopwords[lw] || strings.ContainsAny(lw, "0123456789") {
|
||||
continue
|
||||
}
|
||||
if len([]rune(w)) > len([]rune(best)) {
|
||||
best = w
|
||||
}
|
||||
}
|
||||
if best != "" {
|
||||
add(regexp.QuoteMeta(strings.ToLower(trimWord(best))))
|
||||
}
|
||||
for _, d := range weekdayRe.FindAllString(text, -1) {
|
||||
add(regexp.QuoteMeta(strings.ToLower(d)))
|
||||
}
|
||||
for _, f := range figureRe.FindAllString(text, -1) {
|
||||
add(regexp.QuoteMeta(strings.ToLower(f)))
|
||||
}
|
||||
return pats
|
||||
}
|
||||
|
||||
func trimWord(w string) string {
|
||||
return strings.TrimFunc(w, func(r rune) bool { return !unicode.IsLetter(r) })
|
||||
}
|
||||
|
||||
func excerpt(s string, n int) string {
|
||||
s = strings.Join(strings.Fields(s), " ")
|
||||
r := []rune(s)
|
||||
if len(r) <= n {
|
||||
return s
|
||||
}
|
||||
return string(r[:n]) + "…"
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
Ana: malta, o custo do hosting subiu de 80€ para 120€ por mês
|
||||
Rui: desde quando?
|
||||
Ana: a fatura deste mês já vem com 120
|
||||
Rui: é muito, devíamos ver outros fornecedores
|
||||
Ana: eu peço um orçamento a outro fornecedor até segunda-feira
|
||||
Rita: vamos marcar o jantar de equipa?
|
||||
Rui: sexta às 20h no restaurante O Cantinho?
|
||||
Ana: boa, somos 9 pessoas, eu faço a reserva
|
||||
Rita: lol
|
||||
Tiago: o certificado TLS da API expirou ontem à noite
|
||||
Tiago: já renovei, a API responde outra vez
|
||||
Rui: proponho um alerta no Grafana 14 dias antes de expirar
|
||||
Tiago: eu faço isso esta semana
|
||||
@@ -0,0 +1,8 @@
|
||||
[
|
||||
{"name": "Hosting 80 to 120", "all": ["120", "80"]},
|
||||
{"name": "Quote by Monday", "all": ["orçamento|fornecedor", "segunda"]},
|
||||
{"name": "Dinner Friday 20h", "all": ["jantar", "sexta", "20h"]},
|
||||
{"name": "Dinner place and size", "all": ["cantinho", "9"]},
|
||||
{"name": "Certificate renewed", "all": ["certificado|tls", "renov"]},
|
||||
{"name": "14-day alert", "all": ["14 dias", "alerta|grafana"]}
|
||||
]
|
||||
@@ -0,0 +1,231 @@
|
||||
package tldrbench
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/bwmarrin/discordgo"
|
||||
)
|
||||
|
||||
func write(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(p, []byte(content), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func texts(msgs []*discordgo.Message) []string {
|
||||
var out []string
|
||||
for _, m := range msgs {
|
||||
out = append(out, m.Author.Username+"|"+m.Content)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestLoadAuthorLines(t *testing.T) {
|
||||
msgs, err := LoadChat(write(t, "chat.txt", "Ana: olá a todos\nRui: bom dia\nsegunda linha do Rui\n\nAna: http://x.pt/a?b=1: link\n"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"Ana|olá a todos", "Rui|bom dia\nsegunda linha do Rui", "Ana|http://x.pt/a?b=1: link"}
|
||||
if got := texts(msgs); strings.Join(got, ";") != strings.Join(want, ";") {
|
||||
t.Errorf("got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadClientCopy(t *testing.T) {
|
||||
chat := "Ana — Today at 14:32\nolá pessoal\nvamos ao jantar?\n[14:33]\nque horas?\nRui — Today at 14:35\nsexta às 20h\nAna — 12/10/2026 14:40\nboa\n"
|
||||
msgs, err := LoadChat(write(t, "copy.txt", chat))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"Ana|olá pessoal", "Ana|vamos ao jantar?", "Ana|que horas?", "Rui|sexta às 20h", "Ana|boa"}
|
||||
if got := texts(msgs); strings.Join(got, ";") != strings.Join(want, ";") {
|
||||
t.Errorf("got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadExport(t *testing.T) {
|
||||
js := `{"messages":[
|
||||
{"id":"1","type":"Default","content":"olha isto https://youtu.be/x","author":{"id":"9","name":"ana","nickname":"Ana","isBot":false},"embeds":[{"title":"Um vídeo"}]},
|
||||
{"id":"2","type":"Reply","content":"vi","author":{"id":"8","name":"rui","isBot":false},"reference":{"messageId":"1"},"attachments":[{"fileName":"a.png"}]},
|
||||
{"id":"3","type":"GuildMemberJoin","content":"","author":{"id":"7","name":"x"}},
|
||||
{"id":"4","type":"Default","content":"beep","author":{"id":"6","name":"bot","isBot":true}}]}`
|
||||
msgs, err := LoadChat(write(t, "export.json", js))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(msgs) != 4 {
|
||||
t.Fatalf("got %d messages", len(msgs))
|
||||
}
|
||||
if msgs[0].Author.Username != "Ana" || len(msgs[0].Embeds) != 1 {
|
||||
t.Errorf("first message wrong: %+v", msgs[0])
|
||||
}
|
||||
if msgs[1].Type != discordgo.MessageTypeReply || msgs[1].ReferencedMessage != msgs[0] || len(msgs[1].Attachments) != 1 {
|
||||
t.Errorf("reply wrong: %+v", msgs[1])
|
||||
}
|
||||
if msgs[2].Type == discordgo.MessageTypeDefault || !msgs[3].Author.Bot {
|
||||
t.Error("system and bot messages should be marked so the bot's own filters drop them")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadErrors(t *testing.T) {
|
||||
if _, err := LoadChat(filepath.Join(t.TempDir(), "missing.txt")); err == nil {
|
||||
t.Error("missing file should fail")
|
||||
}
|
||||
if _, err := LoadChat(write(t, "empty.txt", "\n\n")); err == nil {
|
||||
t.Error("empty chat should fail")
|
||||
}
|
||||
if _, err := LoadChat(write(t, "bad.json", "{nope")); err == nil {
|
||||
t.Error("broken json should fail")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvaluate(t *testing.T) {
|
||||
facts := writeFacts(t, `[
|
||||
{"name":"dinner","all":["jantar","20h"]},
|
||||
{"name":"meeting","all":["reuni","terça"]},
|
||||
{"name":"hosting","all":["120","80"]}]`)
|
||||
chat := "Ana: jantar sexta às 20h\nRui: o hosting subiu de 80€ para 120€\nRita: reunião para terça às 11h\n"
|
||||
out := "- Jantar de equipa na sexta às 20h, 9 pessoas.\n- A reunião passa para terça às 11h.\n mais uma linha\n- Hosting subiu para 120€ por mês, 300 pedidos"
|
||||
sc := Evaluate(out, chat, facts)
|
||||
|
||||
if sc.FactsHit != 2 || sc.FactsTotal != 3 || len(sc.Missed) != 1 || sc.Missed[0] != "hosting" {
|
||||
t.Errorf("facts: %+v", sc)
|
||||
}
|
||||
// "9" and "300" are not in the chat; "20", "11", "120" are.
|
||||
if strings.Join(sc.Invented, ",") != "9,300" {
|
||||
t.Errorf("invented = %v", sc.Invented)
|
||||
}
|
||||
if sc.Bullets != 3 || !sc.CutOff || sc.Copied || sc.EnglishIntro {
|
||||
t.Errorf("flags: %+v", sc)
|
||||
}
|
||||
|
||||
if !Evaluate("Here is an overview of the chat:\n- ok.", "", nil).EnglishIntro {
|
||||
t.Error("English intro not detected")
|
||||
}
|
||||
if !Evaluate("- "+strings.Repeat("palavra ", 120)+".", "", nil).Copied {
|
||||
t.Error("one giant bullet should count as copied")
|
||||
}
|
||||
if Evaluate("- Tudo bem.", "", nil).CutOff {
|
||||
t.Error("a finished sentence is not cut off")
|
||||
}
|
||||
// A fact needs all its patterns in one bullet.
|
||||
split := Evaluate("- Jantar sexta.\n- Às 20h há reunião.", "", facts[:1])
|
||||
if split.FactsHit != 0 {
|
||||
t.Error("pattern spread over two bullets must not pass")
|
||||
}
|
||||
}
|
||||
|
||||
func writeFacts(t *testing.T, js string) []Fact {
|
||||
t.Helper()
|
||||
f, err := LoadFacts(write(t, "facts.json", js))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func TestLoadFactsErrors(t *testing.T) {
|
||||
for name, js := range map[string]string{
|
||||
"no patterns": `[{"name":"x","all":[]}]`,
|
||||
"bad regex": `[{"name":"x","all":["("]}]`,
|
||||
"bad json": `nope`,
|
||||
} {
|
||||
if _, err := LoadFacts(write(t, "f.json", js)); err == nil {
|
||||
t.Errorf("%s should fail", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportRender(t *testing.T) {
|
||||
rep := &Report{
|
||||
When: time.Date(2026, 10, 18, 12, 0, 0, 0, time.UTC),
|
||||
Chat: ChatInfo{Source: "chat.txt", Messages: 100, Kept: 60, Chunks: 1, InputChars: 3000},
|
||||
Threads: 8,
|
||||
Facts: []string{"dinner", "meeting"},
|
||||
Models: []string{"m1", "m2"},
|
||||
Variants: []string{"current", "alt"},
|
||||
Runs: []Run{
|
||||
{Model: "m1", Variant: "current", N: 1, Seconds: 60, FirstText: 20, Output: "- <b>x</b> & y", Score: Score{FactsHit: 2, FactsTotal: 2, Hit: []string{"dinner", "meeting"}}},
|
||||
{Model: "m1", Variant: "current", N: 2, Seconds: 70, Output: "- ok", Score: Score{FactsHit: 1, FactsTotal: 2, Hit: []string{"dinner"}, Missed: []string{"meeting"}, Invented: []string{"7"}}},
|
||||
{Model: "m2", Variant: "current", N: 1, Error: "ollama unreachable"},
|
||||
{Model: "m1", Variant: "alt", N: 1, Seconds: 40, Output: "- ok", Score: Score{FactsHit: 0, FactsTotal: 2, Missed: []string{"dinner", "meeting"}}},
|
||||
},
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := rep.WriteHTML(&buf); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
html := buf.String()
|
||||
for _, want := range []string{"2 models, 2 prompts", "1.5 of 2", "failed", "not run", "<b>x</b> & y", "Best on this chat", "<strong>m1</strong>", "missed: meeting", "numbers not in the chat: 7"} {
|
||||
if !strings.Contains(html, want) {
|
||||
t.Errorf("html missing %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(html, "<b>x</b>") {
|
||||
t.Error("output must be escaped")
|
||||
}
|
||||
if strings.Contains(html, "http://") || strings.Contains(html, "https://") || strings.Contains(html, "<script") {
|
||||
t.Error("page must make no network requests")
|
||||
}
|
||||
|
||||
p := filepath.Join(t.TempDir(), "r.json")
|
||||
if err := rep.WriteJSON(p); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
back, err := ReadReport(p)
|
||||
if err != nil || len(back.Runs) != 4 || back.Runs[1].Invented[0] != "7" {
|
||||
t.Errorf("round trip failed: %v %+v", err, back)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSuggestFacts(t *testing.T) {
|
||||
lines := []string{
|
||||
"Ana: bom dia",
|
||||
"Ana: o custo do hosting subiu de 80€ para 120€ por mês",
|
||||
"Rui: vamos ao jantar de equipa na sexta às 20h no Cantinho",
|
||||
"Rita: lol",
|
||||
"Rui: olha isto [link: youtu.be] (preview: \"Tutorial\")",
|
||||
"Tiago: preciso de um backup completo",
|
||||
}
|
||||
facts := SuggestFacts(lines, 10)
|
||||
if len(facts) != 3 {
|
||||
t.Fatalf("want 3 candidates (figures, day+time, link), got %d: %+v", len(facts), facts)
|
||||
}
|
||||
if !strings.Contains(facts[0].Note, "hosting") || facts[0].Note != lines[1] {
|
||||
t.Errorf("first candidate should be the hosting line, got %+v", facts[0])
|
||||
}
|
||||
// Patterns must be usable as they are: they load, and they match the line they came from.
|
||||
js, _ := json.Marshal(facts)
|
||||
loaded, err := LoadFacts(write(t, "skeleton.json", string(js)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, f := range loaded {
|
||||
if sc := Evaluate("- "+facts[i].Note, "", []Fact{f}); sc.FactsHit != 1 {
|
||||
t.Errorf("suggested patterns %v do not match their own line %q", facts[i].All, facts[i].Note)
|
||||
}
|
||||
if len(f.All) == 0 || len(f.All) > 3 {
|
||||
t.Errorf("want 1 to 3 patterns, got %v", f.All)
|
||||
}
|
||||
for _, p := range f.All {
|
||||
if strings.Contains(strings.ToLower(p), "ana") && !strings.Contains(strings.ToLower(f.Note[strings.Index(f.Note, ":"):]), "ana") {
|
||||
t.Errorf("author name leaked into pattern %q", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
if got := SuggestFacts(lines, 2); len(got) != 2 {
|
||||
t.Errorf("max not applied: %d", len(got))
|
||||
}
|
||||
if got := SuggestFacts([]string{"Ana: oi", "Rui: sim"}, 5); len(got) != 0 {
|
||||
t.Errorf("small talk should give no candidates: %+v", got)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user