# The fast router: one token, read from the logits

How `merud` picks a route. This design note sits under
[ARCHITECTURE.md](../ARCHITECTURE.md) and covers step 1 of the
[agent loop](../ARCHITECTURE.md#agent-loop). It belongs to v0.1.

Related: [open question 1](../ARCHITECTURE.md#open-questions) asks whether a 2B model
can route. This note answers it by changing the question — the router stops writing
JSON (JavaScript Object Notation) and starts returning a probability.

---

## The problem

The router runs on every turn and decides one of four things:

| Route | Meaning |
| --- | --- |
| `direct` | answer from the model alone |
| `search` | retrieve from the user's files first |
| `tools` | call MCP (Model Context Protocol) tools |
| `search+tools` | both |

Asking a 2B model to write that as JSON has three failure modes. It emits malformed
JSON and the parse fails. It invents a fifth route. It returns valid JSON with a
confident wrong answer and no way to tell.

The first two stall the turn or force a silent default. The third is worse, because
`meru.turn.*` by route is the measurement that settles open question 1, and a silent
default poisons it.

## The mechanism

A route decision is a classification, so read it from the logits instead of parsing
prose.

1. Prompt the `fast` model with one lettered option per route, each with a
   description, the folders Meru indexes, the tools the user connected, two
   worked examples per route, then the
   session history and the question.
2. Ask Ollama for **one** token with log probabilities, with thinking turned off
   (`think: false`). A thinking model otherwise spends that token starting its hidden
   reasoning.
3. Take the alternatives at position 0, keep the ones whose text is a route letter,
   and turn each log probability back into a probability.
4. Divide each logit by a stored temperature, then normalise so the four sum to 1.
5. The highest is the route. Its probability is the confidence.

The model decodes a single token, so a turn pays for prompt evaluation and nothing
else. The model cannot name a route that does not exist, because the answer is read
from a fixed set of letters rather than from whatever it wrote.

```mermaid
flowchart LR
    Q["question + history"] --> P["build prompt<br/>A–D, folders, tools, examples, then the turn"]
    P --> O["Ollama /api/chat<br/>num_predict 1<br/>logprobs true, think false"]
    O --> T["top_logprobs at position 0"]
    T --> F["keep A–D<br/>exp, temperature, normalise"]
    F --> C{"confidence ≥<br/>min_confidence?"}
    C -- yes --> R["route"]
    C -- no --> FB["search+tools<br/>outcome = low_confidence"]
```

### Why this model

MiniCPM5-2B already fills the `fast` tier in the `lite` profile and `merud` already
keeps it warm. A route decision costs no extra download, no extra memory and no
extra slot in `OLLAMA_MAX_LOADED_MODELS`. Nothing self-hosted is cheaper than a
model that is already resident.

The model name stays in `config.toml`, as every model name does. This design works
with any model Ollama serves; it assumes nothing about MiniCPM beyond a tokeniser
that emits single letters as their own tokens, which the startup probe checks.

## Scope

**In scope.** The route decision, and only that.

**Out of scope, on purpose.** No model rewrites the query. Search uses the question
itself, and on a short follow-up `merud` appends the session's latest earlier question
that isn't only filler words such as "try the last question again". That needs no
model call (see
[How a conversation continues](../ARCHITECTURE.md#how-a-conversation-continues)).
A rewrite would need generated text, a second model call on every search turn;
nobody has measured that search needs one. Skill selection arrives in v0.4 as its own
short call. It could become a second classification, but one decision per call means
one call per decision. Revisit either when there is a number.

This keeps the change small: one new path, one prompt, one parser.

## The prompt contract

The prompt ends with a bare `Answer: ` so the next token is the letter. Options carry
a description, because small models lean on the label text.

```text
<system prompt, when there is one>

Decide how to answer the user's next question.

A = General knowledge, chit-chat, maths, coding or writing help. Needs nothing about the user and nothing recent or live.
B = Look in the user's own saved notes, documents, code repos or past chats. Nothing live, no action. The user's files are in ~/notes, ~/repos/meru; questions about projects kept there, by name, are B.
C = Live, recent or outside data (web, news, scores, weather, prices, email inbox, calendar) or an action (send, book, create, schedule). Nothing from the user's notes. Connected: google (gmail, message, thread, event, drive), obsidian (vault), git-log, web search; questions about these, by name, are C, or D when they also need the user's notes.
D = Needs the user's notes or files AND a live lookup or an action.

Examples:
what's the boiling point of water in Denver -> A
fix the grammar: me and him was late -> A
what did I note about the gym contract -> B
which of my scripts use ffmpeg -> B
is it windy in Chicago now -> C
move my Monday standup to 10 -> C
text Maya the gate code from my notes -> D
does the hotel in my trip notes have rooms free -> D

Conversation so far:
<history, newest first, within the history budget>

Question:
<the user's question>

Reply with one letter and nothing else.
Answer: 
```

The letter-to-route mapping lives in Go, next to the prompt template, so the two
never drift. Do not read the letters from config; they are part of the prompt.

Each part earned its place on the labelled set (see [Calibration](#calibration)):

- **Fixed part first.** The options and examples never change, so they open the
  prompt and the turn's history and question close it. Ollama reuses its work on a
  prompt's opening tokens when the next prompt starts the same way, so the fixed
  part costs little after the first turn. The order also raised accuracy more than
  any other single change.
- **Contrastive descriptions.** Each option names the questions that belong to it
  and says what it excludes. "Email inbox" and "calendar" sit under C, because
  they are live services, not notes.
- **Folders on B's line.** When `[index] folders` lists any, B's line names them
  as `config.toml` writes them, and says that questions about projects kept there,
  by name, are B. `merud` passes them in `Turn.Folders`. Without the line the model
  sent "what database does meru use" to `direct` with 0.902 confidence, because
  nothing told it that "meru" is one of the user's own projects. The folders change
  only when config does, so the line sits in the fixed part and Ollama's reuse
  still works. With no folders the line is left out.
- **Connected tools on C's line.** When the user has connected anything, C's
  line names it and says that questions about these, by name, are C, or D when
  they also need the user's notes. `agent.ConnectedTools` builds the list from
  config: each MCP server and A2A agent that allows a tool, with up to five
  nouns from its allowed tool names; each `[[commands]]` entry by name; and
  "web search" when `web_search` is on with a SearXNG address, or "web pages"
  when only `web_fetch` is. `merud` builds it at startup and on each MCP reload
  and passes it in `Turn.Tools`, so it stays in the fixed part. Without it the
  router sent "what does the onboarding doc in drive say about laptops" to
  `direct` and "when's my next meeting with Priya" to `search`. See
  [Notes from the connected line](#notes-from-the-connected-line).
- **Two examples per route.** None repeats a question in the labelled set, and a
  unit test checks that.
- **Letters A to D in route order.** Every other order tried did worse, and so did
  the words `direct`, `search`, `tools` and `both`, which the tokenizer splits.

The prompt costs about 256 tokens for a question with no history and no folders,
against about 120 before these changes. The eval's three folders add 33 tokens,
and its connected list adds 48 more, for 337.

### Tokeniser detail

A tokeniser may return `A` or ` A` depending on what precedes it. Trim each candidate
token before comparing, and compare case-insensitively. Match on the trimmed text
only, never on a token identifier, because identifiers differ per model.

## Engine changes

The `Engine` interface keeps its four methods. Log probabilities are a request option
and a response field, not a fifth method.

```go
// Options gains two fields. Both default to off, so every existing caller behaves
// as before.
type Options struct {
    // ... existing fields ...

    // LogProbs asks the runtime to report how likely each generated token was.
    LogProbs bool
    // TopLogProbs is how many alternatives to report at each position. Ollama caps
    // this at 20. Zero means "only the chosen token".
    TopLogProbs int
}

// TokenLogProb is one token and how likely the model thought it was, as a natural
// logarithm. Log probabilities are always zero or negative; closer to zero is more
// likely.
type TokenLogProb struct {
    Token   string
    LogProb float64
}

// PositionLogProbs is what the model considered at one position in its answer: the
// token it picked, and the alternatives it weighed.
type PositionLogProbs struct {
    Chosen TokenLogProb
    Top    []TokenLogProb
}

// Completion gains one field, empty unless Options.LogProbs was set.
type Completion struct {
    // ... existing fields ...
    LogProbs []PositionLogProbs
}
```

`OllamaEngine` sets `logprobs` and `top_logprobs` on the request body and copies the
response array across. It converts nothing else; the router owns the arithmetic.

### Ollama version

Log probabilities arrived in Ollama v0.12.11. Meru now needs v0.40.5 or later for
its embedding model, which covers the router too. `merud` reads `/api/version` at
startup and answers no question below it, with a message naming the version it found
and where to update. No compatibility branch, no fallback path — one way to do each thing.

## The router package

`internal/router`. One exported function and one result type.

```go
// Decide picks a route for one turn. It sends a single prompt to the fast model,
// reads the probabilities the model assigned to each route letter, and returns the
// most likely route with its confidence.
//
// It returns an error only when the model call itself fails. A model that answers
// unclearly is not an error: Decide returns the fallback route and says why in
// Outcome, so the caller can carry on and the metrics can record it.
func Decide(ctx context.Context, eng engine.Engine, cfg Config, turn Turn) (Decision, error)

// Decision is what the router concluded.
type Decision struct {
    Route      Route   // the chosen route, or cfg.Fallback
    Confidence float64 // probability of Route after normalisation, 0 to 1
    Outcome    Outcome // ok, low_confidence or degraded
    Probs      map[Route]float64
}

type Outcome string

const (
    // OutcomeOK means the model gave a clear answer above the confidence floor.
    OutcomeOK Outcome = "ok"
    // OutcomeLowConfidence means the winning route scored below min_confidence.
    OutcomeLowConfidence Outcome = "low_confidence"
    // OutcomeDegraded means fewer than two route letters appeared in the
    // alternatives, so the distribution means nothing.
    OutcomeDegraded Outcome = "degraded"
)
```

### The arithmetic

```go
// probs turns the model's log probabilities into a distribution over routes.
//
// Ollama reports a natural logarithm per token. math.Exp turns each back into a
// probability. Dividing the logit by a temperature above 1 flattens the
// distribution. `make router-eval` fits the value; with the prompt above,
// MiniCPM5-2B needs 1.25. Temperature never changes which route wins, only how
// sure the router claims to be.
func probs(pos engine.PositionLogProbs, temp float64) map[Route]float64 {
    raw := map[Route]float64{}
    for _, alt := range pos.Top {
        r, ok := routeForLetter(alt.Token) // trims and lowercases
        if !ok {
            continue // not one of our letters; ignore it
        }
        raw[r] = math.Exp(alt.LogProb / temp)
    }
    sum := 0.0
    for _, v := range raw {
        sum += v
    }
    if sum == 0 {
        return nil // caller reports OutcomeDegraded
    }
    for r := range raw {
        raw[r] /= sum // normalise so the routes sum to 1
    }
    return raw
}
```

A route letter missing from the alternatives counts as zero. With four letters and
`top_logprobs = 20`, all four appear on any sane prompt; fewer than two appearing
means something is wrong with the prompt or the model, which is what `degraded`
records.

## Config

```toml
[router]
top_logprobs     = 20             # Ollama's cap
temperature      = 1.25           # fitted by `make router-eval`; see Calibration
min_confidence   = 0.45           # below this, take the fallback route
fallback         = "search+tools" # the safe superset: retrieval and tools both allowed
decision         = "top"          # or "marginal"; see Decision rules
search_threshold = 0.30           # "marginal" only
tools_threshold  = 0.25           # "marginal" only
```

`merud` validates all seven at startup: `top_logprobs` between 1 and 20,
`temperature` above 0, `min_confidence` between 0 and 1, `fallback` one of the four
routes, `decision` either `top` or `marginal`, and both thresholds between 0 and 1.

`search+tools` is the fallback because it is the only route that cannot make the turn
fail for want of context. It costs more, which is the point — an unsure router should
spend tokens rather than guess.

## Calibration

`make router-eval` scores the router against the local Ollama. It needs the fast
model pulled, and skips otherwise.

**The labelled set.** `internal/router/testdata/routes.jsonl` holds 167 questions
written the way a user types them, 34 to 48 per route, each with its route and a
reason. 18 carry `"tag": "connected"`: they ask for mail, the calendar,
Drive, the Obsidian vault, a local git command or the clock. Some are follow-ups that carry a `history` of earlier turns. The harness
gives every row the same folders, `~/notes`, `~/repos/meru` and
`~/repos/blog` (`evalFolders`), so a row can ask about "meru" by name the way
a user asks about their own project. A fixed rule splits the set: of every ten rows
with the same route, the 3rd, 6th and 9th go to a held-out set of 48 rows, and the
other 114 form the fit set. The last 9 carry `"tag": "images"`: they ask about
pictures in the folders, such as a receipt to read, and route `search`, where
`view_image` rides with the other file tools; one asks about pictures in
general and routes `direct`. The measurements below ran on the first 153 rows. New rows go at the end of the file, so the rows already
there keep their side of the split. Tune on the fit set and
report the held-out numbers.

**The report.** The harness sends every question through `Decide` with the shipped
config and keeps each raw completion. It then replays the arithmetic offline, so one
pass over the model covers every temperature and threshold. For each set it prints
accuracy (the route `Decide` returned) and top-pick accuracy (before any fallback),
per-route precision and recall, the confusion matrix, mean confidence when right and
when wrong, expected calibration error (ECE) over ten bins, the fallback rate, the
share of turns that *missed* (the route lacks a search or a tool the question needs;
a fallback never misses) and latency at p50 and p95.

**Temperature.** A sweep from 0.5 to 5 picks the value with the lowest ECE over all
rows. Temperature never changes the top pick, so fitting it on every row leaks
nothing into the accuracy numbers. The held-out set alone, at 46 to 48 rows, gives an ECE
too jumpy to fit on.

**Threshold.** A sweep of `min_confidence` shows the fallback rate, accuracy and
missed rate at each floor. Pick the floor with the highest accuracy on the fit set,
and break ties by the lowest missed rate.

**Refit** after any change to the prompt, the examples or the fast model, and write
the results into `config.toml` defaults and `config.example.toml`.

**Writing help that says "my".** On 4 November 2026 the router sent
`rewrite "Meru streamlines my workflow by handling my tasks efficiently."` to
search at 0.518 against 0.425 for direct. Three rows joined the labelled set for
it: two rewrites from that chat, the second with the first as its history, and a
proofread of quoted text that says "my". With them the set scored 0.890 on the
fit set and 0.776 held out. Two prompt changes routed both rewrites direct, and
both cost more than they gained:

| Change | Fit set | Held out |
| --- | --- | --- |
| none | 0.890 | 0.776 |
| option A names "writing help on text the question gives, such as rewrite or translate, even when that text says I or my" | 0.805 | 0.735 |
| the direct example "fix the grammar: me and him was late" becomes "…me and my boss was late to my review" | 0.839 | 0.714 |

The first change lowered search recall on the fit set from 0.941 to 0.765. The
prompt stays as it was, and the agent loop sends such a question direct with a
rule of its own (ARCHITECTURE.md, "Routing", and `writingOnText` in
`internal/agent/rewrite.go`). The rows stay in the set, so the score shows what
the router alone gets right.

## Decision rules

`decision` picks how `decide` turns the four probabilities into a route.

- **`top`**, the default, takes the most likely letter. A winner below
  `min_confidence` takes the fallback, `search+tools`.
- **`marginal`** asks two yes/no questions. B and D both search, so
  P(search) = P(B) + P(D). C and D both call tools, so P(tools) = P(C) + P(D). The
  turn gets a search when P(search) reaches `search_threshold`, and tools when
  P(tools) reaches `tools_threshold`. Two yeses give `search+tools`, one gives
  `search` or `tools`, and none gives `direct`. It uses no floor, so a spread-out
  distribution adds what the spread covers rather than both. Fewer than two letters
  still take the fallback, with outcome `degraded`. The outcome is otherwise `ok`,
  so the metric's outcome set stays the same three values. The confidence is the
  weaker of the two answers: P(search) for a yes and 1 − P(search) for a no, and
  the same for tools.

The case behind `marginal` came from a real turn: "what was the last email I sent?"
gave direct 0.049, search 0.473, tools 0.260, search+tools 0.218. `top` chose
search and the turn got no Gmail tools, though the two tool letters held 0.478
between them.

**How the harness compares them.** `make router-eval` replays both rules over the
same saved completions at the configured temperature, 1.25. For each set it prints
exact accuracy, the missed rate (the route lacks a search or tools the label
needs), and two over-offer rates: a search the label doesn't need, and tools the
label doesn't need. It prints them apart because extra tools cost more: each
tool's schema lengthens the prompt and slows the turn, and the `lite` model picks
worse from a long tool list. It also counts the rows whose route differs from
`top`'s.

It tries each threshold in {0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5}, 49 pairs, and
ranks them on the fit set alone: a pair must keep its tool over-offer within 5
points of `top`'s, and among those the lowest missed rate wins, then the highest
accuracy, then the lowest tool over-offer, then the lowest search over-offer. When
the fit set can't tell two pairs apart, the higher tools threshold wins, then the
higher search threshold, because the pair that offers less costs less on questions
the set lacks. The held-out numbers play no part in the pick.

**Results.** Measured on the development machine: Ollama 0.34.0, MiniCPM5-2B at
Q4_K_M, 107 fit rows and 46 held-out rows, `min_confidence = 0.45`. Two runs gave
the same routes on every row; only latency moved. The fit set picked
`search_threshold = 0.30` and `tools_threshold = 0.25`. Rates are shares of all
rows in the set.

| rule | set | accuracy | missed | extra search | extra tools | rows changed |
| --- | --- | --- | --- | --- | --- | --- |
| `top`, floor 0.45 | fit | 0.850 | 0.103 | 0.084 | 0.019 | 0 |
| `top`, floor 0.60 | fit | 0.804 | 0.084 | 0.121 | 0.037 | 8 |
| `marginal` 0.30 / 0.25 | fit | 0.841 | 0.065 | 0.112 | 0.019 | 7 |
| `top`, floor 0.45 | held-out | 0.761 | 0.196 | 0.130 | 0.000 | 0 |
| `top`, floor 0.60 | held-out | 0.717 | 0.109 | 0.174 | 0.065 | 7 |
| `marginal` 0.30 / 0.25 | held-out | 0.739 | 0.152 | 0.174 | 0.022 | 7 |
| `top`, floor 0.45 | connected rows | 0.500 | 0.333 | 0.444 | 0.000 | 0 |
| `marginal` 0.30 / 0.25 | connected rows | 0.444 | 0.278 | 0.556 | 0.000 | 3 |

On the held-out set `marginal` missed 7 of 46 turns against 9, got one row fewer
exactly right (34 against 35), added an unneeded search to 8 turns against 6, and
added unneeded tools to 1 against 0. Raising `top`'s floor to 0.60 misses less
still, but offers unneeded tools on 3 held-out turns against `marginal`'s 1. The best pairs on the fit
set:

| search / tools | fit missed | fit accuracy | held-out missed | held-out accuracy | held-out extra tools |
| --- | --- | --- | --- | --- | --- |
| 0.30 / 0.25 | 0.065 | 0.841 | 0.152 | 0.739 | 0.022 |
| 0.30 / 0.20 | 0.065 | 0.841 | 0.109 | 0.761 | 0.022 |
| 0.25 / 0.25 | 0.065 | 0.832 | 0.152 | 0.717 | 0.022 |
| 0.35 / 0.25 | 0.075 | 0.850 | 0.174 | 0.717 | 0.022 |
| 0.30 / 0.30 | 0.075 | 0.841 | 0.152 | 0.761 | 0.000 |

The first two pairs score the same on the fit set. The tie rule picked 0.25; 0.20
would have missed two fewer held-out turns, but the pick can't see those.

Over all 153 rows, the chosen pair routes 14 differently from `top`. Seven
improve. Four of them now match the label, such as "try the last question again"
(direct to search) and "look up reviews for the books on my reading list" (search
to search+tools). Three `tools` requests that `top` sent to search, among them the
email question and "text alex that I'm on my way", now get tools along with an
unneeded search. Six get worse: four `direct` or `tools` questions gain a search
they don't need, "what documents do you have indexed" gains tools, and "what's
changed in upstream ollama since the version my project pins", which `top` sent to
the fallback, loses its tools. "What does the onboarding doc in drive say about
laptops" misses its tools under both rules and gains a search under `marginal`.

**The connected-server rows.** `top` sent 6 of the 18 somewhere that lacks what
they need, and `marginal` sent 5. Both rules send "search my obsidian vault for
notes about sourdough" and "add a line to today's daily note in obsidian" to
`search`, with P(tools) at 0.07 and 0.14, and "what's uncommitted in my blog repo right
now?" to `search` at 0.92. No threshold on the grid reaches those. In the eval,
without the system prompt and history of the real turn, the email question gave
search 0.646, tools 0.111, search+tools 0.165: `top` chose search, and `marginal`
chose search+tools with P(tools) at 0.276, just over the 0.25 bar. At the time
the router's prompt took no list of connected servers, so the eval gave it none;
the agent loop's rules after the router handled server names and tool nouns.
[Notes from the connected line](#notes-from-the-connected-line) has the numbers
with the list.

**Decision.** `marginal` lowers the missed rate on both sets and keeps tool
over-offer within 3 points, so it ships behind `decision = "marginal"`. The
default stays `top`. On 46 held-out rows the gain is 2 turns, it costs one exact
match, and the email question that prompted it already gets Gmail tools from the
fifth rule after the router (ARCHITECTURE.md, "Routing"). Revisit with labelled
turns from real transcripts.

## Observability

The `meru.route` span already exists. Add these attributes:

| Attribute | Value |
| --- | --- |
| `meru.route.decision` | the chosen route |
| `meru.route.confidence` | the winning probability |
| `meru.route.outcome` | `ok`, `low_confidence` or `degraded` |

Spans take floats; metrics do not. Add one counter and keep its attributes bounded:

| Metric | Type | Attributes | Answers |
| --- | --- | --- | --- |
| `meru.route.decisions` | counter | route, outcome | how often each route wins, and how often the router is unsure |

Never put the question, the prompt or the probability map on a metric attribute. The
probability map belongs on the span, and only when `capture_content = true`, because
it leaks something about the question.

## Tests

Standard `testing`, table-driven, no assertion library.

- **Scoring.** `eval_test.go` checks the split, ECE, percentiles and the report
  against fixtures. The real-model harness in `eval_integration_test.go` runs only
  with the `integration` tag.
- **`probs` arithmetic.** Table of alternative lists and temperatures against expected
  distributions. Include a list with no route letters, one with a single letter, and
  one where two letters tie.
- **Letter matching.** `A`, ` A`, `a`, ` a` all map to the same route. `AB`, `Alpha`
  and `1` map to nothing.
- **Fake Ollama.** `httptest` server returning a canned `logprobs` array. Assert that
  the request body carries `logprobs: true`, the configured `top_logprobs` and
  `num_predict: 1`. No test needs a real model.
- **Outcomes.** Each of `ok`, `low_confidence` and `degraded` reached by a fixture,
  with the fallback route returned for the last two.
- **Version gate.** A fake `/api/version` below v0.40.5 makes `merud` answer no
  question, with a message naming the version and where to update.

`go test -race ./...` must pass.

## Done when

`merud` routes a turn from a single decoded token, the dashboard shows
`meru.route.decisions` by route and outcome, and the routing call adds under 150 ms to
a warm turn on the development machine.

## What this changes in ARCHITECTURE.md

Per [AGENTS.md](../AGENTS.md), ARCHITECTURE.md changes first and the level 200 and 100
pages follow in the same pull request.

1. **[Agent loop](../ARCHITECTURE.md#agent-loop), step 1** — say the router returns a
   route by classification, and nothing else.
2. **[Who decides what](../ARCHITECTURE.md#who-decides-what)** — the row for the route
   decision gains "reads the probability of each route letter from one decoded token".
3. **[Engine layer](../ARCHITECTURE.md#engine-layer)** — note the two new `Options`
   fields and the `Completion` field, and that the interface stays at four methods.
4. **[Model tiers](../ARCHITECTURE.md#model-tiers)** — record that the `fast` tier now
   needs a runtime that reports log probabilities, and name the Ollama version.
5. **[Metrics](../ARCHITECTURE.md#metrics)** — add `meru.route.decisions`.
6. **[Open questions](../ARCHITECTURE.md#open-questions)** — rewrite question 1. The
   question is no longer whether a 2B model can emit a valid route, but whether its
   probabilities separate the four routes well enough to act on.

## Coding notes this work must add

- `docs/coding-notes/router.md` — what the package does and why a probability beats
  parsed text.
- `docs/coding-notes/engine.md` — update for the new option and result fields.
- `docs/coding-notes/go-basics/maps.md` — first use of a map with a named key type.

## Open items

- Refit the temperature and the threshold on labelled turns from real transcripts
  once there are enough of them.
- Questions about the Obsidian vault and a repository's working tree route to
  `search` at 0.68 to 0.92 under both decision rules (see "Decision rules"). The
  prompt names the indexed folders but no connected servers; naming them in
  option C is the next thing to measure.
- `tools` has the weakest recall: the model still answers questions about recent
  events, such as last night's score or the latest release, from memory, and sends
  requests such as "text alex that I'm on my way" to `search`. In v0.2 the agent
  searches on the `tools` route as well, so a question about the user's files that
  lands there still gets excerpts.
- Measure whether skill selection should become a second classification. Needs a
  number first.

## Notes from the v0.1 build

- **Endpoint.** Ollama 0.34 returns log probabilities from `/api/chat`, so the
  router uses it rather than `/api/generate`, and shares the chat code path with
  every other call.
- **Thinking.** The engine sends `think: false` whenever `LogProbs` is set.
  Models that don't think accept the flag and ignore it.
- **Spellings.** When both `A` and ` A` appear among the alternatives, their
  probabilities add up. A tie goes to the earlier letter.
- **First measurement.** On the development machine, MiniCPM5-2B at temperature
  1.0 took about 13 ms per warm decision and picked the expected route for 2 of 7
  hand-picked questions, leaning towards `search+tools`. Calibration and prompt
  tuning are the next steps (see open question 1 in ARCHITECTURE.md).

## Notes from the v0.2 calibration

Measured with `make router-eval` on the development machine: Ollama 0.34,
MiniCPM5-2B at Q4_K_M, held-out set of 36 questions.

| | v0.1 prompt, T = 1.0 | v0.2 prompt, T = 1.25 |
| --- | --- | --- |
| accuracy (after fallback) | 0.444 | 0.806 |
| top-pick accuracy | 0.472 | 0.778 |
| recall: direct, search, tools, search+tools | 0.22, 0.11, 0.89, 0.67 | 1.00, 0.78, 0.44, 0.89 |
| missed (route lacks a needed search or tool) | 0.167 | 0.056 |
| fallback rate at `min_confidence = 0.45` | 0.056 | 0.139 |
| ECE | 0.222 | 0.150 (0.043 over all 121 rows) |
| mean confidence right / wrong | 0.70 / 0.66 | 0.85 / 0.48 |
| prompt tokens, no history | 116 | 256 |
| latency p50 / p95 | 56 ms / 64 ms | 28 ms / 49 ms |

- **The lean.** The v0.1 prompt sent almost everything to `tools` or
  `search+tools`; top-pick accuracy on the fit set was 0.34, and 2 of 22 `direct`
  questions came back `direct`. The new prompt spreads the four routes out. Its
  errors now lean the other way: questions about recent events come back
  `direct`.
- **What helped.** Top-pick accuracy on the fit set, one change at a time: the
  v0.1 prompt scored 0.34; contrastive descriptions alone, 0.42 (and lower on the
  held-out set); eight examples after the question, 0.61; the fixed part moved
  first, 0.75 to 0.89 depending on the wording. Of the wordings in that last group
  the one shipped had the lowest log loss on the fit set. Three other letter orders
  scored 0.40 to 0.55, the labels `direct`/`search`/`tools`/`both` 0.47, and the
  digits 1 to 4 0.73.
- **Temperature.** The v0.1 prompt needed T = 4 for its lowest ECE. With the new
  prompt the raw probabilities sit close to calibrated, and ECE stays flat from 0.9
  to 1.25, so the fit is a light touch.
- **Latency.** The new prompt is twice as long and still faster. The likely
  reason: Ollama reuses the fixed opening from the previous turn and evaluates
  only the history and the question.
- **Full profile.** The `full` profile uses the same fast model, so these numbers
  cover it too.

## Notes from the folder line

Measured with `make router-eval` on the development machine: Ollama 0.34.0,
MiniCPM5-2B at Q4_K_M, temperature 1.25, `min_confidence = 0.45`. The labelled set
grew from 121 to 135 rows. The 14 new rows are 11 `search` questions and 3 `direct`
look-alikes. The `search` rows ask about the user's projects by name ("what
database does meru use", "what files can you see") or ask to retry ("try the last
question again", "search again, I think it's in there"). The look-alikes ask the
same kind of thing about someone else's project: "what database does wordpress
use", "what is sqlite good for", "what language is kubernetes written in".

| | old prompt, 121 rows | old prompt, 135 rows | folders on B's line, 135 rows |
| --- | --- | --- | --- |
| fit top-pick accuracy | 0.859 | 0.811 | 0.874 |
| fit accuracy (after fallback) | 0.859 | 0.800 | 0.884 |
| held-out top pick | 28 of 36 (0.778) | 30 of 40 (0.750) | 32 of 40 (0.800) |
| held-out search recall | 0.778 | 0.667 (8 of 12) | 0.917 (11 of 12) |
| held-out tools recall | 0.444 | 0.444 | 0.444 (4 of 9) |
| fallback rate, fit / held-out | 0.035 / 0.139 | 0.042 / 0.125 | 0.011 / 0.025 |
| ECE over all rows | 0.043 | 0.061 | 0.065 |

- **Two placements that lost.** A separate line after the options that said
  questions about those projects are "B or D" pushed 10 fit `search` rows to
  `search+tools`, and held-out top pick fell to 29 of 40. The same line saying
  only "B" pulled requests to text or send into `search`: 28 of 40. The shipped
  version adds the folders to B's own line.
- **What got worse.** Held-out `tools` requests now go to `search` rather than
  `search+tools`: 4 of the 9, such as "text alex that I'm on my way". The router
  also falls back less, and a fallback never misses, so the held-out missed rate
  rose from 0.056 to 0.175. In v0.2 the
  agent searched on `tools` as well, and no route called tools, so these misses
  cost nothing then. From v0.3 a request sent to `search` gets no tools, unless it
  names a tool server (see the second rule below).
- **Temperature.** The sweep's lowest ECE over all rows is now at T = 1.10
  (0.050), against 0.065 at 1.25. The gap is small, so the default stays at 1.25.
- **A rule after the router.** Even with the folder line, the router sent "what
  database does Meru use to store its index?" to `direct` at 0.621, and the model
  made up an answer. So the agent loop, outside this package, changes a `direct`
  route to `search` when the question names an indexed folder as a whole word,
  such as "meru" for `~/repos/meru`. A wrong guess costs one search of about
  50 ms. The `meru.route` span and `meru.route.decisions` still record what the
  router chose; the turn's route event, log line and span show `search`. See
  [docs/coding-notes/agent.md](coding-notes/agent.md).
- **A second rule, for tools (v0.3).** The router sent "Search my Obsidian vault
  for notes mentioning 'AI'" to `search` at 0.65. With no tools offered, the
  model said it couldn't search the vault. So after the folder rule, the agent
  loop adds tools when the question names a connected MCP server or A2A agent as
  a whole word and the route offers none: `direct` becomes `tools`, and `search`
  becomes `search+tools`. The names come from the tools dispatch offers, such as
  "obsidian" from `obsidian.obsidian_simple_search` and "research" from
  `a2a.research.summarize`, read on each turn because `configure` can add a
  server while merud runs. Built-in tools don't count: their owner is "meru",
  the assistant's own name. With the rule, the same question listed the vaults,
  searched one and answered. The router's span and metric keep what it chose,
  as with the folder rule.

## Notes from the connected line

C's line now names what the user has connected (see
[The prompt contract](#the-prompt-contract)). Measured with `make router-eval` on
the development machine: Ollama 0.34.0, MiniCPM5-2B at Q4_K_M, temperature 1.25,
`min_confidence = 0.45`, the same 153 rows split 107 fit and 46 held out. Every
row gets the eval's folders and the connected list in `evalTools`, which is what
`agent.ConnectedTools` gives for the catalog's `google` and `obsidian` servers
with their allow lists, a `git-log` command and web search:

```text
google (gmail, message, thread, event, drive), obsidian (vault), git-log, web search
```

Three runs of each prompt gave the same route on every row; only latency moved.
"Before" is the same harness with `Turn.Tools` left empty. Rates are shares of
the rows in the set. "Missed" means the route lacks a search or tools the label
needs; "extra tools" means the route offers tools the label doesn't need.

Held-out set, 46 rows:

| rule | prompt | exact match | missed | extra search | extra tools |
| --- | --- | --- | --- | --- | --- |
| `top`, floor 0.45 | before | 0.761 (35) | 0.196 (9) | 0.130 | 0.000 |
| `top`, floor 0.45 | after | 0.739 (34) | 0.152 (7) | 0.109 | 0.065 |
| `marginal` 0.30 / 0.25 | before | 0.739 (34) | 0.152 (7) | 0.174 | 0.022 |
| `marginal` 0.30 / 0.25 | after | 0.783 (36) | 0.065 (3) | 0.130 | 0.065 |
| `marginal` 0.50 / 0.30, fit-set pick | after | 0.739 (34) | 0.152 (7) | 0.087 | 0.065 |

Every row, and the 18 connected-server rows, `top` rule:

| set | prompt | exact match | missed | extra search | extra tools |
| --- | --- | --- | --- | --- | --- |
| all 153 | before | 0.824 (126) | 0.131 (20) | 0.098 | 0.013 |
| all 153 | after | 0.856 (131) | 0.098 (15) | 0.052 | 0.033 |
| connected 18 | before | 0.500 | 0.333 | 0.444 | 0.000 |
| connected 18 | after | 0.722 | 0.278 | 0.167 | 0.000 |

The model's top pick, before the confidence floor, matched 36 of 46 held-out
rows against 34, and `tools` recall on the held-out set rose from 5 of 13 to 9
of 13. The floor now sends 5 held-out rows to the fallback against 3, which is
why exact match under `top` fell by one row while the top pick gained two.

- **What got better.** Fifteen rows now match their label under `top`, among
  them "what was the last email I sent?", "how many unread emails do I have",
  "when's my next meeting with Priya" and "find the offsite agenda doc in my
  google drive" (search or search+tools to tools), "what does the onboarding doc
  in drive say about laptops" and "how much is a tesla model 3 these days"
  (direct to tools), and "what is the default chunk size in meru's indexer"
  (direct to search). "Text alex that I'm on my way" and "looks good, send it"
  moved from search to search+tools: not exact, but no longer missing tools.
- **What got worse.** Ten rows that matched no longer do. Seven are
  search+tools questions that name mail, a calendar or a message and now go to
  tools: "email the team the release checklist from my meru repo", "is the
  meeting from my trip notes on my calendar yet?", "forward the summary we made
  yesterday to my manager", "post the release notes from my blog changelog to
  slack", "did anyone email me about the bug I described in my notes", "search
  the web for fixes to the error in my debugging log" and "what's changed in
  upstream ollama since the version my project pins". The agent loop searches
  on every route but `direct`, so a `tools` turn still gets the user's files;
  these cost less than the "missed" column says. The other three gained a
  fallback or tools they don't need: "what did we talk about yesterday about
  the router?", "what documents do you have indexed" and "train leaves at 3:40
  and the trip is 2h35m, when do I get in?".
- **What the list doesn't fix.** "Search my obsidian vault for notes about
  sourdough" and "add a line to today's daily note in obsidian" still go to
  `search` at P(search) 0.81 and 0.61: the word "note" pulls them to B. "What's
  uncommitted in my blog repo right now?" goes to `search` at 0.95, because
  nothing in "git-log" says "uncommitted".
- **Wordings tried.** Each wording was scored on the fit set with `top`, and
  the one with the highest fit-set exact match ships. Some of them, with
  fit-set exact match and missed rate: before, 0.850 and 0.103.
  "Connected: …; questions about these, by name, are C" with no word about D:
  0.813, missed 0.140; it pulled search+tools questions to C. The list alone,
  with no instruction: 0.813, missed 0.178. The shipped sentence with "web
  pages" and "remember" in the list too: 0.850, missed 0.093. A C line that
  opens "Use a connected tool (…)" and a D line that says "AND a connected
  tool": 0.710. The list on a line of its own after the options: 0.841. The
  shipped sentence with the list of servers, commands and web search: 0.907,
  missed 0.075.
- **Refitting.** The temperature sweep's lowest ECE on the fit set sits at
  T = 1.25, the shipped value, so it stays. The confidence floor gives the same
  fit-set routes at every value from 0 to 0.45, so it stays at 0.45. The grid
  for `marginal` picks 0.50 / 0.30 on the fit set, where it ties 0.30 / 0.25 on
  missed (0.065) and wins on exact match (0.888 against 0.850). On the held-out
  set 0.50 / 0.30 misses 7 against 3. `marginal` is off by default, the fit set
  can't separate the two pairs on the number the rule ranks first, and the
  held-out set favours the shipped pair, so the thresholds stay at 0.30 / 0.25.
- **The rules after the router.** Replayed over the 153 top-rule routes with
  the eval's tools: the folder rule fires once, on "suggest a title for a blog
  post about running LLMs locally", where it adds a needless search; before the
  list it also rescued "what is the default chunk size in meru's indexer",
  which the router now gets. The server rule fires on the two Obsidian rows
  above and gives them their tools, as before. The noun rule fires on 5 rows
  against 8; before, it rescued four email and Drive questions, and now three
  of them route to tools on their own, leaving "reply to the last email from my
  landlord saying thanks". The remember and web rules fire on no labelled row,
  with or without the list, because the set has no question they match. All
  five stay.
- **Cost.** The list adds 48 prompt tokens, from 289 to 337 with the eval's
  folders. It changes only when config does, so Ollama reuses its work on it,
  and warm decisions stayed at about 30 ms.

## Notes from web first

The fourth rule after the router now reads a phrase list instead of three
words, and a URL counts too (ARCHITECTURE.md, "Routing" and "Web first").
The router and its prompt didn't change, so the calibration stands. We ran the
new tests over the 153 labelled questions without a model:

- **Phrases.** The list matches 4 questions against 2 for "web", "internet"
  and "online". The two new ones, "look up reviews for the books on my reading
  list" and "look up the current price of everything on my shopping list
  note", carry the label `search+tools`, so the rule can only add tools the
  label wants. No question holds a URL.
- **Names.** The detector finds a name in 12 questions. It found 15 before
  the four-word cap on quoted names, which drops sentences quoted to rewrite
  or fill in. Three of the 12 sit on `direct`, where the named search never
  runs, and four say "my" or point at mail. That leaves five that could search
  the web when the files don't mention the name: "Lisbon" in the weather
  question and "SFO" in the drive question, which want the web; "Sintra" and
  "TODOs", which the user's notes likely cover; and "shipping v0.2 today" in
  "post … to slack", a message that would reach SearXNG for nothing.

## Notes from the web on every route

Every route, `direct` included, now offers `web_search` and `web_fetch` while
`web_search` is on (ARCHITECTURE.md, "Who decides what"). A real question drove
it, here with an invented song: "what does the song maname maname sung by r.
devi acvtually mean" routed `direct` at 0.966. The route fits the label the
eval would give it, general knowledge, but the model knew the song only in
part, had no web tool, and told the user it couldn't search the web. On a
`direct` turn the model now has the web tools and a line in the tools note
that tells it to call `web_search` when it isn't sure of a fact, so a factual
question the router sends `direct` can still look things up.

The router and its prompt didn't change, so the calibration stands. Of the 153
labelled questions, 36 carry the label `direct`: general knowledge, chit-chat,
arithmetic, writing help and the like. Each such turn now carries two more tool
schemas and one more line in the prompt, and the model decides whether to
search. The router needs no change: a `direct` label still means "no files
and no connected tool", which stays true.
