# Benchmark

> scripts/bench.sh [sessions] [turns] [steady|burst] measures newt under load with no model provider and no API key.

Source: https://newt.julianbouchard.com/benchmark/

Synced from `docs/benchmark.md` in the newt repository.

`scripts/bench.sh [sessions] [turns] [steady|burst]` measures newt under load with no model provider and no API key. It needs Postgres (`NEWT_DATABASE_URL`, or `docker compose up -d --wait` for the repo's dev database) and runs everything else itself.

## What runs

- **The fake model** (`newt-loadtest fake-model --cache-scope routed --machines 4`) speaks the OpenAI Responses stream. Its timings and tool use come from the profile of a production assistant, about 16 tool calls a turn. Prompt caching is simulated per `prompt_cache_key` and spread over four machines once a key passes 15 requests a minute, as OpenAI routes, so a key that is too busy or a rewritten history misses as it would against the real thing. A prompt is readable from the cache 1 second after it was sent: against OpenAI on 2026-11-12 a second request with the same prefix hit the cache every time it started 1.1 s or more after the first.
- **The mock app** (`newt-loadtest mock-app`) is a context provider with a production-sized prompt and workspace, and an MCP server whose tools have that assistant's latencies and error rates.
- **newt**, built in release mode, with default runner settings (`cache_key_buckets = "auto"`).
- **The driver** (`newt-loadtest run`) follows each turn on the live feed. In the default `steady` scenario sessions start at random over 60 seconds and a person pauses 20 to 90 seconds between messages, so each session makes a few model calls a minute, as people do. The `burst` scenario starts every session at once and sends the next message 2 seconds after a turn ends: about 20 model calls a minute per session, past what one cache key holds, and every turn start at the same moment. It is the worst case for the database pool and the cache, not what people do.

## What it reports

- From the driver: time to the read receipt, time until the turn starts, turn duration, model calls and tool calls per turn, the cached share of input, and failed turns.
- newt's CPU time and resident memory at the end of the run, and how many warnings it logged.
- The cache report card from `newt-report`: the cached share of all input, and every cache miss with its cause, including a session moved to another cache key.
- The fake model's cache log (`cache.jsonl` in the run directory): one line per request with its cache key, the machine it went to and the key's home machine, the key's requests in the last minute, and how many leading prompt blocks that machine held, could read, or did not have.
- Time spent waiting for a database connection when accepting an event, claiming the session and loading history, in `newt-report --summary`.

Turn duration is dominated by the fake model's simulated latency and the mock tools, so it says whether newt adds waiting, not how fast a real model is. The numbers that are newt's own are time to read receipt, time to turn start, the cached share and the miss causes, and CPU and memory per session.

## Reading a result

Run it on a quiet machine: other load shows up as newt CPU time and as slower Postgres. Use a fresh database: the same code read a p90 receipt of 1,080 ms on a long-used dev database and 307 ms on a new one. Compare runs of one commit against another at the same session count, not against numbers from another machine. A failed turn, a miss labelled anything but `first call of the session`, or warnings in newt's log are worth a look before any timing.

## Results

200 sessions, 3 turns each, on an Apple M5 Pro (15 cores, 24 GB) with Postgres 18 in OrbStack, on 2026-11-12. The same machine ran the fake model, the mock app and the driver.

| | `69f29b5` | `0152f99` |
|---|---|---|
| Turns completed | 600 of 600 | 600 of 600 |
| Model calls | 3,879 | 3,887 |
| newt CPU time | 1:56 | 0:12 |
| newt resident memory | 25 MiB | 26 MiB |
| Read receipt p50 / p90 | 3 / 1,080 ms | 4 / 285 ms |
| Event to turn start p50 / p90 | 12 / 2,044 ms | 13 / 586 ms |
| Turn duration p50 / p90 | 16.7 / 28.2 s | 16.3 / 27.4 s |
| Cached share of input | 76.0% | 77.9% |
| Misses: first call of a session | 200 | 200 |
| Misses: routing | 117 | 117 |
| Warnings in newt's log | 0 | 0 |

What changed between the two:

- **CPU.** At `69f29b5` every live feed and running turn woke for every notification on the instance; a profile showed most of newt's time waiting on that one channel's lock. `0673c06` wakes only the session a notification is for.
- **The p90 read receipt and turn start** come from the first turn, when all 200 sessions post in the same moment. After the first turn both are under 15 ms in both runs. `0152f99` opens every database connection at boot instead of growing the pool during that burst.
- **Routing misses** happen in the first minute, while auto cache keys grow from one key to about 130 and sessions move to new keys, each of which the fake model routes to another of its four machines. After the first 80 seconds there are almost none. `18cc08f` raised the key cap from 64 to 256 so that at this rate (about 1,600 calls a minute) each key stays under OpenAI's ~15 a minute; the fake model's four machines cannot show that difference.

Turn duration is the fake model's and the mock tools' simulated latency, as above. One MCP call timed out after 90 seconds in a run before `0152f99`: the mock app's latency tail, not newt.

## Per milestone

The same harness (HEAD's fake model, mock app and driver), newt built at each milestone commit, a fresh database per run, on 2026-11-12. The oldest and newest commits ran twice to show the noise. Score: the mean of four parts, each 0 to 100: turns completed; read receipt p90 (100 ms to 3 s) and turn start p90 (250 ms to 5 s); cached share of input (50% to 90%); CPU per model call (2 ms to 50 ms) and memory per session (0.25 MiB to 5 MiB). Latency, CPU and memory are scored on a log scale between those ends.

| Commit | Milestone | Score | Read receipt p90 | Turn start p90 | Cached | newt CPU per model call |
|---|---|---|---|---|---|---|
| `d7e6e89` | Runtime complete | 74.9 / 71.0 | 280 / 330 ms | 864 / 2,243 ms | 80.5 / 80.9% | 28.0 / 24.5 ms |
| `4a77802` | Hot path | 77.0 | 227 ms | 576 ms | 80.3% | 29.3 ms |
| `81e6a76` | Shared cache keys | 73.9 | 250 ms | 569 ms | 75.9% | 29.7 ms |
| `2c7c472` | Hardening | 70.7 | 410 ms | 789 ms | 75.9% | 29.9 ms |
| `0f992bb` | MCP 2026-07-28 | 75.0 | 225 ms | 481 ms | 75.9% | 29.7 ms |
| `69f29b5` | MCP-native agent | 72.2 | 307 ms | 716 ms | 75.7% | 28.9 ms |
| `80d8a89` | Wakeup fix, embedded Postgres | 84.9 / 85.5 | 217 / 195 ms | 499 / 519 ms | 77.7 / 78.4% | 3.1 / 3.1 ms |

All 600 turns completed in every run. The features of six milestones left the score between 71 and 77; the wakeup fix in `0673c06` is the one change that moved it. A move of 3 points or less is within the noise. In this simulator sending no cache key (before `81e6a76`) caches more than shared keys; a replay against the real provider showed the reverse, so the cached share here reflects the fake model's four machines.