Skip to content
Trazum

1.9.0

The error band, measured

The release that found out the central claim was false, and fixed it.

±15% had been printed on every report for eight releases, with every dollar figure descending from it, and nothing in this repository established that it held. The first run of measure-token-band.mjs against the official counting endpoint found it did not: the numeric sample was 30.6% under, Spanish prose 22.1% under. Nine of eleven samples underestimated, always in the direction that under-reports cost.

Two things were wrong. Digits were counted at three per token where Claude splits them far more finely — corrected in isolation, that sample went to -5.0%. And the estimator turned out to be calibrated for English specifically, not for prose: German measured -37.3% under one divisor that served every language.

The band is measured now, at ±15%, and it landing back on the old number is a coincidence rather than a restoration: that 15 bounded nothing, and this one bounds twenty-one samples across seven languages and six text types, worst case 11.2%. Every language divisor has a held-out test in a different register.

Also here: trazum baseline and a check that gates on drift rather than only on a ceiling; a pull-request comment that leads with what the branch costs; and below-cache-minimum no longer asserting from an estimate near a hard threshold, which was wrong advice rather than an imprecise figure.

The sections below are as they accumulated, entry by entry, and were not consolidated: they are the record of what happened in the order it happened.

Fixed#

SECURITY.md claimed prompts are never stored, and that stopped being true when --suggest --cache shipped. The sentence was written when it was unqualified and correct; the cache arrived later and nobody went back to the security document. It writes the model's raw response, keyed by the prompt, to a file under the user's home directory.

Nothing about the feature is wrong — it is opt-in, it is local, and the files are 0600 in a 0700 directory precisely because a prompt is the most sensitive thing this tool touches. What was wrong is a security document telling a reader that no such file exists, which is the kind of error that survives review because it reads as reassurance.

The paragraph now says what is guaranteed (nothing about a prompt reaches a server that keeps it) separately from what the CLI can be asked to do on the machine it already runs on, and names where to delete it.

The repository-hardening checklist listed a step that is done and enforced. Pinning every third-party action to a commit SHA was item 6, phrased as something to run once there is network access. It has been done for several releases and security.test.js fails any uses: naming a tag or a branch — so it is not a checklist item, it is an invariant, and it moved to a section that says so. The list also had two items numbered 3 and a closing line counting five of six.

Changed#

The corpus went from eleven samples to twenty-one, and every divisor now has a held-out test. The eleven that set the band left three languages calibrated on a single sample each, which is a fit rather than a measurement: a divisor chosen to minimise the error on one file will always look good on that file.

Ten samples were added, in two rounds. The first round gave Italian, Portuguese and Dutch a sample each — three languages that had a divisor by inheritance and no evidence — and the second gave every calibrated language a second sample in a different register: the first set are support prompts, the second are code-review prompts, different vocabulary and different length. That second sample is what turns the first from a fit into a finding, and all seven held:

                calibrated on   held out on
english             +1.0%          +0.4%
german              -9.2%          -8.5%
french              -1.2%          -5.8%
spanish             -6.2%          -9.7%

The divisors moved as a result — Italian and Dutch had been taking English's 4 — and the corpus-wide worst case is unchanged at 11.2%, on Japanese, which no divisor touches. Nothing in twenty-one samples is outside ±15%.

Italian had to be rebuilt, and the reason is worth recording. Its first function-word list was half Spanish: per con del una sempre are as common in one as the other, so they earned nothing, the margin rule tied, and an Italian code-review prompt came back null, fell through to the English divisor and measured -21.9%. The detector was working exactly as designed — it refuses rather than guesses — and the fault was a word list that could not tell the two apart. The replacement is words Italian has and Spanish does not. Over-correcting it broke the prose sample instead, which is the shape of this whole file: a list tuned on one register is a list tuned on one register.

What a hundred samples per language would have bought, and why they were not written. The counting endpoint is free, so the constraint was never money. It is that every sample here was written by the same hand, and ninety more of those is ninety more of the same bias — a tighter-looking number resting on nothing new. Twenty-one real prompts bound the band honestly; a hundred invented ones would bound it decoratively. The corpus grows one sample at a time now, and the samples worth adding are the ones that came from somebody's actual work.

Fixed#

below-cache-minimum was asserting from an estimate, and near the threshold that made it wrong advice. It compares the stable prefix against a hard limit — 512 tokens on Claude Opus 5 — and then tells the reader caching will not work here. The prefix is estimated, so a prompt measured at 505 tokens could really be at 540 and cache perfectly well. Not an imprecise figure: a reader told to stop looking at the single largest saving Trazum offers.

Near the line it now says so, and names the way to settle it — --exact-tokens, against an endpoint that is free. Far below the line it stays quiet, because a hedge on every case is a hedge nobody reads.

And it does not hedge a number the caller measured. count defaults to estimateTokens; a caller who supplied their own counter has an authoritative figure, and telling them it might be wrong is its own kind of dishonesty — it pushes them toward a check they have already done.

Both directions mutation-tested: hedging always fails the exact-counter test, hedging never fails the near-threshold ones.

This was listed as known, not fixed one release ago. The window shrank when language detection took the worst estimate from −37.3% to −9.2%; it did not close, which is why this is a fix rather than a note.

Fixed#

The estimator was calibrated for English and silently wrong for every other Latin language. Measured across eleven samples: German −37.3%, Spanish −22.9% and −22.1%, French −15.1%, English +1.0%. Nine of eleven underestimated, always in the direction that under-reports cost. Characters per token says why — English 3.44, French 2.66, Spanish 2.53, German 2.02 — while one divisor of 4 served all of them.

estimateTokens now detects the language and divides accordingly:

german-prose        -37.3%  →   -9.2%
spanish-unaccented  -22.9%  →   -6.5%
spanish-prose       -22.1%  →   -6.2%
french-prose        -15.1%  →   -1.2%
worst in corpus      37.3%  →   11.2%   (Japanese, untouched)

The published band drops to ±15% — the measured worst case rounded up, the same rule that briefly made it 25. Landing back on the number that was a guess for eight releases is a coincidence, not a restoration: that 15 bounded nothing, and this one bounds eleven samples across four languages and six text types.

The signal is not accents, and that was tested rather than assumed. A Spanish sample with zero diacritics measured −22.9% against −22.1% for accented Spanish, which killed the hypothesis the previous release recorded. What separates these languages is which words they are made of, so language.ts counts function words — the of and to against der die und ist against que los las del.

It answers null when unsure, and most of its tests are about earning that. A three-line prompt, a JSON schema, English instructions wrapped around a Spanish example: no answer is safe for any of them, and a wrong language applies another language's divisor to text that does not want it. null falls back to the English divisor, which is what the estimator always did. Two bars guard it — four distinct function words minimum, and a 1.6× margin over the runner-up — and removing either one fails the suite.

One caveat stated plainly rather than buried: the four Latin divisors are calibrated on one or two samples each, so their residuals are in-sample and optimistic by construction. The band is set by the seven samples nothing was fitted to. The honest test is the next held-out sample in Spanish, French or German, and the corpus grows one sample at a time now.

Known, not fixed#

below-cache-minimum compares an estimated prefix against a hard 512-token threshold, so an underestimate can report "caching will not work here" when it would — wrong advice, not just an imprecise figure, and it costs the reader the largest saving Trazum offers. The worst estimate on measured text is now −9.2% rather than −37.3%, which shrinks the window considerably but does not close it. The advisory should hedge near the threshold; that is its own change.

Added#

The corpus can grow now, and three samples were added to falsify a hypothesis. Measuring the band left one finding unexplained: Spanish prose comes out 22.1% under while English comes out 1.0% over, and accents are not the cause — weighting them from 2 to 5 moves the figure three points. The candidate explanation is merge-table coverage: text that is not English costs more tokens per character whatever its diacritics.

spanish-unaccented.txt is the test that can prove that wrong. It is Spanish with zero accented characters, so if accent density were a usable detector for "this is not English" — and on the old corpus it separated perfectly, 0.00% against 1.71% — this sample would slip past it and stay underestimated. If instead it lands where accented Spanish lands, the phenomenon is the language and the accents were a coincidence of one file. french-prose.txt (1.83% accented) and german-prose.txt (1.64%) say whether other Latin languages behave like Spanish or like English.

Adding them required fixing something first. The freshness digest covered the whole corpus, so it could not tell an edited file from an added one and answered both with "re-run the script" — correct for an edit, wrong for an addition, because it retires eight measurements that cost an API call each to admit one new sample. The corpus was effectively frozen: growing it was gated on a key nobody wanted to spend.

Digests are per sample now, via digestOfOne, and the two cases get what each deserves. A file that changed since it was measured fails — its measurement describes different text, which is the dangerous case because it passes while being wrong. A file with no measurement skips out loud and is named, with the command to run, because a gap in coverage is something to report rather than a reason to distrust what has been measured. The existing fixture was migrated without new API calls, which is sound only because the whole-corpus digest still matched: that match is the proof those eight files are the ones that were measured.

One guard had to be loosened to its intent rather than its letter: it asserted import { digestOf } literally and failed the moment digestOfOne was imported alongside it — a guard that breaks when you use more of the thing it protects.

A global correction factor was tried and rejected. The best available (×1.05) takes the worst case from 22.1% to 18.2%, and does it by pushing Japanese from +11% to +17% and English from +1% to +6%. That is redistributing error, not reducing it, and it damages the two samples the estimator gets right.

Fixed#

The ±15% error band was never true, and now it is measured. Eight releases printed it on every report and every dollar figure descended from it, while nothing in this repository established that it held. The first run of scripts/measure-token-band.mjs against the official counting endpoint found two of eight samples outside it, both underestimating:

numeric-heavy    estimated 277, actual 399   -30.6%
spanish-prose    estimated 352, actual 452   -22.1%

Underestimating tokens means under-reporting cost. Trazum was telling people their prompts were cheaper than they are — the flattering direction, and the worst one for a tool whose whole argument is honest cost accounting.

One constant was simply wrong. Digits were counted at three per token; Claude splits long runs far more finely, because a merge table cannot cover every number. Correcting it in isolation — nothing else touched — takes the numeric sample from −30.6% to −5.0% and moves no other sample more than four points.

The Spanish error is not about accents, and that matters. Weighting accented characters from 2 to 5 moves that sample by three points. Spanish words tokenize into more tokens than English words of the same length even when they are pure ASCII, because Spanish is thinner in the merge table — so no per-character-class constant can fix it, and one Spanish sample cannot calibrate a signal that would. English prose lands at +1.0%, which says the estimator's structure is sound and its coverage of non-English text is not.

So the published band is now ±25%: the measured worst case (22.1%) rounded up, because eight samples cannot bound a worst case tightly and the honest direction to be wrong in is the pessimistic one. It is ESTIMATE_ERROR_BAND_PCT, exported from the core — it had been a literal in twenty-three files with its only machine-readable copy in a test, which is why correcting it meant a hand sweep across three locale catalogues, four READMEs, the MCP tool descriptions, the web app and the demo. A guard now fails when any file states a band the code does not publish, with CHANGELOG.md excluded because rewriting history to match the present is the opposite of a changelog.

The roadmap predicted CJK would be the problem. CJK is fine at −3.2% and +11.2%. It was wrong about which text type and right that one number cannot cover all of them.

Changed#

1.8.0 is on npm. @trazum/core, @trazum/cli and @trazum/mcp were published by hand on 2026-08-13. The first publish had to be manual — a trusted publisher is configured on a package's settings page, and that page does not exist until the package does — so every release after this one goes through a tag with no credential anywhere.

The documentation caught up: the "not published yet" notes are gone from RELEASES.md, the front page and both package READMEs, and Getting started leads with npx @trazum/cli instead of instructions for building from source, which are now folded away for people working on Trazum itself.

The guard that watched those notes was keying on the wrong thing, and the publish proved it. It asked whether v1.8.0 was tagged, on the reasoning that release.yml publishes on a tag and nothing else — checkable offline, which a test in CI should be. But the first publish could never go through a tag, so no tag was pushed, and the repository went on telling every visitor that nothing was installable while three packages sat on the registry. That is the second signal this claim has outlived; the first asked whether the changelog had a heading for the manifest version, which is a release cut here rather than a package on npm.

There is no third proxy. Publication does not reverse, so the assertion is one-directional now: no file may claim nothing is published. docs/releasing.md records both traps the real publish hit — a 404 Not Found - PUT is npm hiding an auth failure behind a missing-scope error, and npm view 404s for minutes after a successful publish because the packument propagates behind the tarball.

Fixed#

npm was silently rewriting the manifest of both published binaries. Every npm publish answered "bin[trazum]" script name was cleaned — npm stripping the ./ from "./dist/index.js" and uploading a manifest that differs from the one in this repository. On npm 12 the same correction reads "was invalid and removed", which would put a package declaring a bin and carrying no executable on the registry: npx @trazum/cli would resolve and then do nothing.

Caught on the first real publish attempt, in the wall of npm notice lines nobody reads during the one command this repository cannot take back. Both manifests now say dist/index.js, which npm has nothing to correct, and a guard asserts it for every publishable workspace — plus that the target actually travels in the tarball, since a bin pointing outside files is the same defect arriving by a different route.

Fixed#

The CLI test suite passed in CI and failed on a contributor's laptop. Seven tests, the first time anybody ran npm run verify on a machine whose locale is not English — which was during a release, on the maintainer's Mac, with LANG set to es_ES.UTF-8. Three spawns in i18n.test.js built their environment inline from process.env and asserted on English output, so they inherited whatever the machine said. A CI runner leaves LANG unset, so the bug was invisible to the only place that was looking.

Five variants of that environment object had grown across the test files by then. Three of them cleared LANG, LC_ALL and TRAZUM_LOCALE; none cleared LC_MESSAGES, which detectLocale also reads — so even the files following the "correct" pattern were one variable away from the same failure.

There is one environment now, in packages/cli/test/env.mjs, and it clears the list of variables imported from the detector rather than a copy of it. LOCALE_ENV_VARS is exported from packages/cli/src/i18n/index.ts and the detector maps over it, so the list is the implementation: a variable added there is read by the detector and cleared by the tests in the same commit, or in neither.

It clears rather than pinning TRAZUM_LOCALE: 'en', which was the first attempt and was wrong — that outranks the project config, so every test taking its language from "locale": "es" in trazum.config.json would have reported in English and asserted against the wrong catalogue. Clearing the environment leaves the precedence chain intact and only removes the machine from it.

Two guards keep it: one fails when any test file builds a spawn environment inline, naming the file, and one asserts the shared environment clears every variable the detector reads. Both mutation-tested, including reverting a file to the exact shape that carried the bug.

Added#

The pull-request comment leads with what the branch costs. The Action has been posting a budget table since it shipped: a list of files and whether each one fits its ceiling. Useful, and not the question a reviewer is holding. What a pull request proposes is a change, so the comment now opens with the change — against the baseline recorded on the base branch — and puts the ceiling check underneath it.

> [!CAUTION]
> **This branch adds 64 tokens (+67.4%) to the prompts here** — over the limit of 0 tokens, 5%.

| | Prompt | Baseline | Now | Change |
|:--:|---|--:|--:|--:|
| 🆕 | prompts/triage.md | – | 64 | +64 |

Monthly cost **$129.75 → $132.95** (+$3.20)

Only the directions that cost money are itemised. A list of every file that shrank buries the two rows somebody has to act on — though a branch that made things cheaper still gets its headline, because that is worth saying.

The rendered block comes from the same outcome the exit code was computed from, so a green comment and a red build cannot disagree. It shouts only when a threshold was actually crossed, names every limit that was crossed rather than the first, and refuses to print a monthly delta when the scenario or the price list moved — a figure in a pull-request comment gets quoted in a meeting, and two different measurements subtracted is not a saving.

No change to the Action was needed. It already posts whatever check --markdown-out writes, so the report carrying the cost diff is the whole mechanism.

Added#

trazum baseline, and a check that fails on drift rather than only on a ceiling. budgets answers "does this file fit". That is a ceiling, and a ceiling has a blind spot: a repository sitting at 95% of every budget passes forever while a pull request adds four hundred tokens across a dozen files. Nothing busted, bill up. A baseline answers the other question — did this get worse than the commit we agreed on — and it is the twelfth command.

trazum baseline [dir] records what the estate costs now to a file you commit. With a baseline block in trazum.config.json, trazum check on a directory reads it and gates on it with no flag, because a gate you have to remember to pass an argument to runs in the author's terminal and not in CI. --no-baseline skips it for one run.

The behaviour the whole thing turns on: a prompt that is new counts. Comparing only the paths present in both documents would let a five-thousand-token addition through every threshold — it is in neither the grown list nor the baseline total — so the demonstration case is a run where every budget is green, no existing file grew, and the build fails anyway because somebody added a file.

The threshold is in tokens, and the money is only reported. A dollar figure comes from the token count, the usage scenario and the price list, and two of those move for reasons that have nothing to do with the prompts. A baseline holding dollars would fail a build the day a model was repriced, calling a price change a regression, and a gate that cries wolf is a gate somebody deletes. When the scenario or the price list has moved, the report says which one instead of subtracting two different measurements and presenting the difference as a saving.

A declared-but-missing or corrupt baseline fails the run. A gate the config asked for and could not execute is not a pass; otherwise deleting one file silently switches CI off. That includes a hand-edited totals.tokens that disagrees with its own per-file counts — the corruption that otherwise looks completely normal. Neither threshold has a default and omitting both is a config error, because every default here is silently wrong: zero tolerance gets the block deleted within a week, and a generous one is a gate passing things nobody agreed to.

trazum.baseline.json's format joins the frozen API in VERSIONING.md, and it is the strongest of those promises because the file is committed: it outlives the Trazum that wrote it, so the document carries a version and an unknown one is a loud error rather than a best-effort read.

Internally, the directory walk that decides what counts as a prompt was extracted from checkDirectory so both commands share it. Two walks would be two definitions of the estate, and the baseline would end up recording files the gate never checks.

Changed#

Trazum was selling the weakest half of itself. The front page led with "cut what your prompts cost", and directly underneath it the demo — real output, not a mock-up — showed the rules recovering $24.00 a month while a single advisory sitting above them was worth $528.40. Twenty-two times more. The tool has been telling the truth about that gap for several releases; the pitch had not caught up.

The framing now leads with the finding rather than the trim. The headline is that most of an LLM bill is not the prompt, the advisories come first in What it actually does with the trimming after them, and the caption under the demo points at the two lines that make the argument instead of hoping the reader notices. The package descriptions, the web app's title, tagline and lede all say the same thing in both locales.

Nothing about the product changed — no rule, no advisory, no number. This is the description catching up with what was already being measured.

The two counts the pitch now rests on — thirteen advisories, twelve rules — are asserted against RULES and the AdvisoryId union, so a fourteenth advisory cannot ship while the front page still says thirteen.

Fixed#

The zero-dependency invariant was documented for three packages and enforced for two. security.test.js looped over a typed list — packages/core and packages/cli — while SECURITY.md credited the invariant to all three published packages. packages/mcp hand-rolls its JSON-RPC layer for exactly this reason, and the test named as the reason had never heard of it. The list is derived from the root workspaces globs now, with an assertion that the derivation found the MCP server at all, so a suite that quietly resolves to nothing cannot report "0 failures" from having checked nothing. CONTRIBUTING.md's first rule said "the core and the CLI" and now says what is actually enforced, including why apps/web is exempt.

Nothing checked that the README's images exist. A moved or renamed asset renders a broken-image placeholder to every visitor on the front page and tells the person who moved it nothing. Local src and srcset paths are now asserted to exist, absolute URLs deliberately excluded — somebody else's uptime is not something a test here can hold.

The README's layout diagram also described apps/web as "Optimise and Compare", one tab short since the prompt library shipped.

Added#

A real screenshot of the web app on the README, light and dark, swapped by prefers-color-scheme so it matches the theme the reader is already in. Captured from the production build rather than drawn, on the same wordy support prompt the CLI demo uses.

Changed#

The web app got a display voice, two tiers of depth, and browser surfaces that belong to its own palette. It had been correct and characterless: shadcn's shapes wearing Trazum's colours, one system sans doing every job from the wordmark to the percentage that is the entire point of the page, and a dark theme in violet-grey borrowed from every other dashboard. Nothing about the identity changed — paper, terracotta and ink are exactly what they were — but the page now looks like it was built rather than assembled.

Fraunces Variable is self-hosted from npm and carries the wordmark and the figures; the CSP's font-src 'self' holds, so no font is fetched from a foreign host at runtime. Elevation is now two tiers and only two: working surfaces get a hairline and a breath of shadow, and the result panel — the page's one focal moment — drops its border rather than drawing the same edge twice. The dark palette was warmed to the same umber cast as the light one so the terracotta sits on Trazum's paper in both schemes. Selection, caret, focus ring and scrollbars are themed from the palette, and the body sets tabular figures, because every number on this page is a measurement and measurements align or they wobble.

Two defects went with it. The reorder callouts were drawing a 3px coloured rule down one side — a border doing a highlight's job — and are now a terracotta wash. And on a narrow screen the history card sat between the Optimise button and the answer the reader had just pressed it for; the two column wrappers collapse to display: contents below lg, which makes every card a grid item and lets history take order-last without moving a line of markup.

Added#

The README leads with the receipts, and every markdown file caught up with the code. The front page now opens with a transcribed — not mocked — optimize run as an SVG terminal, down to the closing line where Trazum admits the advisory is worth 22× what the rules saved. The first draft of that SVG contained exactly one invented figure, which its own header comment forbids; it was caught and replaced with the real third advisory. A new guard derives the front page from the workspace manifests: the architecture diagram had silently omitted @trazum/mcp for the whole day that package existed, and now the next package added has to appear in the README or publish.test.js fails.

The sweep also caught: SECURITY.md crediting zero dependencies to "the core and the CLI" when three packages now carry the invariant; CONTRIBUTING.md counting three workspaces and describing phrases.ts as "Spanish (and, in time, other)" seven languages later; docs/releasing.md publishing two packages when the workflow publishes three; the CLI README's command table listing five commands of eleven; and VERSIONING.md freezing every API surface except the newest one — the MCP server's tool names and input schemas are now part of the promise. RELEASES.md and ROADMAP.md gained the account of everything that shipped under 1.8.0's banner since their last update, provider layer and measurement layer both.

Automatic recovery from container rollbacks, at scripts/recover-workspace.sh and a Claude Code SessionStart hook. The remote environment this repository is developed in restored its container disk to a stale snapshot more than twenty times across two working sessions — every tracked file reverted, mid-work, silently, always to the same commit. The first draft of this very script was destroyed by the failure it exists to repair, one commit short of being safe.

A script inside the repository cannot prevent that, and this one does not claim to: it reverts along with everything else. What survives a rollback is the remote, so recovery is always fetch, reset to origin/main, reinstall — and the script makes those one safe move. The rollback signature is precise (HEAD strictly behind origin/main) and everything else is refused: a tree that is ahead is work in progress, a diverged tree is a choice no script should make (exit 1), and uncommitted changes are stashed by name before any reset rather than discarded.

.claude/hooks/session-start.sh runs it at the start of every Claude Code on the web session — and only there, guarded on CLAUDE_CODE_REMOTE, because on a local machine the tree is the developer's own and resets are not a hook's call.

Eleven behavioural tests drive the real script against real git repositories in temp directories, including the one that matters most: the rollback also reverts .git's remote-tracking refs, so a script that compared HEAD to origin/main's ref would see them equal and announce nothing to recover. The fixture builds exactly that state, and only a real fetch passes it. Thirteen mutants, thirteen killed — among them "the stash disappears", "an ahead tree also gets reset" and "a push appears", each the difference between a recovery script and a data-loss tool with a reassuring name.

The definitive fix is platform-side — recreating the environment so it stops restoring a stale snapshot — and is not something a repository can do to itself.

Added#

trazum prune <file> --cases <file> — which few-shot examples earn their tokens, measured rather than guessed. The eleventh command.

The redundant-examples advisory asks a textual question: does this example look like an earlier one? This asks a stronger one: does removing it change any answer? Two examples can be textually unalike and teach the same thing, and the few-shot section is routinely most of a prompt.

Leave-one-out against the prompt's own noise floor. Ask the full prompt twice to learn how much the model disagrees with itself, then remove one example and ask again; a removal that moves the answer less than that did no observable work. The thresholds come from evaluate's verdictFor rather than a second set.

The only command that asks before spending. The bill is (2 + examples) × cases — 220 calls for a nine-example prompt over twenty cases. Without --yes it prints the figure and stops, and it prints it before looking for a provider, so the cost is visible without a key configured. plannedCalls is exported and pure, and a test asserts the number it promises is the number spent.

It reports "no effect on these inputs" and never "delete this." An example may exist for a case the given inputs do not contain. Nothing is edited, and the strength of the claim is bounded by the inputs — which only the caller can judge.

withoutExample locates blocks by position rather than by prompt.replace(text, ''): a copy-pasted few-shot section contains identical blocks, and a text replace would match the first occurrence for both, so measuring the removal of the second would describe the removal of the first.

Twelve mutants, twelve killed. Two defects came out of it that no test would have found, because both were about reading the output:

  • The first draft duplicated agreement from evaluate.ts as a bag-of-words F1 while that one is Jaccard over normalised text — two different numbers under one name, with a comment in the copy claiming they were the same measure. agreement and pooled are now exported and shared, which makes the comment true.
  • The report put a green tick beside "0% agreement without it", which reads as approval next to the one line meaning "leave this alone", and printed Example: as every block's identifying line. Both only visible by running it against a local stand-in provider, which is how they were found.

@trazum/mcp — Trazum as an MCP server, so an agent can price and budget a prompt before it sends it. Three tools over stdio: check_prompt, optimize_prompt, list_models. It runs on the caller's machine, one process spawned by the client exactly like the CLI — no service to host, and no prompt leaves the machine.

check_prompt has three outcomes rather than two, which is the reason it exists: inside budget, over budget but the rules would fix it, or over budget with content that has to be cut. A boolean throws away the actionable half.

What it cannot do is the design. No paths — every tool takes text, and the package imports @trazum/core rather than @trazum/core/node, so the file-reading capability is absent rather than unused. No network: --suggest and eval are deliberately not exposed, because a tool an agent can invoke in a loop must not be able to spend the caller's money. No writes.

The JSON-RPC layer is written by hand, and that was not the first attempt. It used @modelcontextprotocol/sdk and thirteen tests passed against a real process. Then publish.test.js refused it: every publishable package here carries no runtime dependencies outside the repository, and the reason security.test.js gives — every dependency is somebody else's code running on untrusted text — applies to an MCP server with more force than anywhere else in Trazum. Relaxing the invariant at the point it matters most would have been backwards, so the invariant won.

What that costs is stated in the module: the implementation covers initialize, notifications/initialized, tools/list, tools/call and ping, and answers anything else with -32601. It has been driven by a raw newline-delimited client in the tests, not by every MCP client in existence.

Writing it by hand immediately produced the bug that justifies testing it: the notification check sat inside the method switch, listing the two notifications/* methods by name. A notification is defined by the absence of an id, not by its method, so {"jsonrpc":"2.0","method":"initialize"} with no id got a reply — a protocol violation some clients tolerate and others hang on, which is the worst kind because it works in testing. A test that asked for the rule rather than for the two names found it.

Seventeen mutants: fifteen killed by tests, two by the compiler. Four new guards in publish.test.js and the release workflow had to be updated too — a third publishable package needs a README, a LICENSE, provenance, and a publish step ordered after @trazum/core, and every one of those was a test failure rather than something anybody remembered.

A .pre-commit-hooks.yaml, for teams who manage hooks with pre-commit. scripts/pre-commit stays the recommended path; this is for the repositories that already have a .pre-commit-config.yaml, which in practice means Python shops whose prompts live in .py string literals.

Two hooks rather than one with a flag: trazum-check is a gate and fails the commit, trazum-doctor never does. Collapsing them would let a --survey argument silently turn a gate into a report.

It needs the first npm publish to work, and says so. The trazum executable comes from additional_dependencies rather than from installing this repository, and the two alternatives were tried rather than reasoned about: installing the repo root gives Executable trazum not found, because the root is a private workspace root with no bin; adding a bin plus a prepare that builds gives Workspaces not supported for global packages, because pre-commit installs with npm install -g.

The mechanism is verified with locally packed tarballs standing in for the registry — the gate fails a prompt over budget, passes one inside it, and the survey hook exits 0. The registry lookup is the only untested part.

An advisory for a schema the request could carry instead of the prompt. The one finding here that is not a trade-off.

A prompt that spells out its output shape in a fenced block pays for it in input tokens on every call, and gets the weaker of the two available guarantees for the money: prose asks the model to comply, a response schema makes the decoder comply. output_config.format, response_format, responseSchema — every major API takes the same shape as a request parameter. Moving it is cheaper and stricter.

Reported, never done. It is not a change to the prompt but to the code that sends the prompt, and a rule that deleted the schema would leave a prompt asking for a shape it no longer describes, sent by a client nobody updated — strictly worse than the prompt it started from. A test asserts the schema and its fences survive --level aggressive.

The one way it could do harm, and what stops it. A fenced JSON block is either an output contract, which moves for free, or data a few-shot example needs, which breaks the prompt if moved. Nothing guesses: a block counts only when a phrase from the new OUTPUT_CUES_BY_LANGUAGE appears in the 240 characters before it. A schema with no cue is left alone; a prompt in a language the dictionaries do not cover raises nothing at all — a false negative, which is the right direction to be wrong in, and stated as one rather than papered over.

The cue is matched through normalizeForCompare, so FORMATO DE SALIDA — and formato de salida: are the same phrase, and it is quoted back verbatim from the prompt rather than translated: it is the author's text, not the report's.

The figure is attached and the uncertainty is in the words. Trazum knows how many tokens the block holds, which is reproducible; it cannot know from here whether a given provider offers the parameter. Withholding the number for that reason would be the wrong trade — it is right if the move is available — so the advisory says plainly that it does not check. The same posture as model-downgrade, which carries a figure and admits to being a keyword heuristic.

Thirteen mutants, thirteen killed. One found a real defect: the first draft filtered out keys shorter than three characters, copied from the restated-format detector where it stops a two-letter key matching a word in prose. Nothing is matched against prose here, so all it did was undercount schemas whose fields are called id or ok. Deleting it changed no test, which is what a line with no reason looks like; it is gone, with a test for short field names in its place.

Added#

trazum doctor finds preambles that could share a cache entry and do not. The first finding in this repository that no single prompt can produce.

Prompt caching is a byte-for-byte prefix match, so twelve prompts assembled from the same system preamble — identical except that one has a trailing tab, another reordered two bullets, and a third writes E-Commerce where the rest write e-commerce — occupy twelve cache entries and share nothing. Every one of those files is individually fine, which is exactly why no per-prompt analysis finds it.

drift says which kind of work it is: whitespace means the text already agrees and a formatter fixes it; wording means somebody has to pick one.

Three refusals, and they are the design:

  • Grouped by the first block. Caching matches from the start of the request, so prompts whose opening paragraphs differ share nothing however identical the rest is. Grouping on a later block would name prompts that can only be made to share a prefix by reordering their instructions — the one transformation this repository keeps out of aggressive for being dangerous.
  • Gated on the model's own cacheable minimum, via a new exported cacheableMinimum. A model whose caching is unknown — what the live pricing overlay assigns to one it has never seen — yields Infinity, so nothing is reported. Telling somebody to unify a preamble across twelve files to enable caching their provider may not offer spends their afternoon, and unlike a wrong number on a report nothing later corrects it. The same directory against Haiku 4.5, minimum 4,096, produces nothing for a 1,398-token preamble.
  • Prompts already byte-identical are not reported. They share an entry today.

No dollar figure, and that is a finding rather than a gap. The saving lives in the cache hit rate, and cacheHitRate is an input to the cost model rather than something it derives — --cache-hit-rate applies one value to every prompt, so the model has no term for how many distinct cache entries exist. Pricing this would mean inventing how the calls are spread across the group, which is the one thing here only the operator knows. A test asserts structurally that no field on the result looks like money.

Thirteen mutants, eleven killed. The two survivors are equivalent rather than uncovered — two independent guards cover the same case, so no test can distinguish them — and both are documented in the test file so nobody removes one believing the other carries the weight.

Fixed#

A suite crashed on every clean checkout and exited 0. token-band.test.js read fixtures/ with an unguarded readdirSync, and that directory does not exist until somebody runs scripts/measure-token-band.mjs. So it threw ENOENT during suite construction, node's runner printed the stack as a diagnostic, reported fail 0 and exited 0 — which is precisely what the top of that file forbids: "'0 failures' from a check that measured nothing is the most misleading thing a suite can report." The skip beneath it was written for a directory that exists and holds no per-provider file; it never covered the directory being absent, which is this repository's normal state. Found while diagnosing an unrelated verify failure.

Fixed#

The README claimed prompts are never stored on any server. They are, once the prompt library is switched on: trazum_prompt_versions.text holds the text of every saved version, and has since the library shipped. The library is off by default, so the sentence was true in the configuration everybody develops in and false in the one an operator opts into.

That is the same shape as the Content-Security-Policy that blocked analytics nobody had enabled, found the same day — and worse in one respect. A broken policy eventually breaks visibly for the operator who enabled it. A privacy sentence is read once, by somebody deciding whether to trust the thing, and nothing ever tells them it was wrong.

The section now states both configurations and names the column. Three tests keep it honest: the schema is asserted still to store prompt text, so if that ever stops being true the claim can go back to being absolute and a failing test says so rather than the guard going quiet; the exact sentence that was wrong may not reappear; and both configurations have to be identified by the thing that selects them. docs/accounts.md was already accurate and is now linked from here.

Five mutants, five killed, the first of them the original defect put back verbatim.

The new Content-Security-Policy blocked analytics, silently. connect-src 'self' shipped in the same change as the nonce, and Analytics.tsx posts to https://eu.i.posthog.com. An operator setting NEXT_PUBLIC_POSTHOG_KEY got a page that rendered perfectly and sent nothing, with the reason visible only in a browser console nobody was reading.

Nothing caught it because the key is unset in CI and in development: the configuration where it breaks is the one no test exercises. Found by reading the policy next to the component while reviewing an unrelated posthog-js bump — not by any check in this repository.

The host now comes from lib/analytics, which both files read, so the policy and the request cannot name different hosts again. With no key the policy is byte-for-byte what it was; with one, connect-src gains exactly one origin.

It is an origin, never the configured string. A policy is built by joining text with ;, so a host of evil.test; script-src * would not have widened connect-src — it would have appended a directive of somebody else's choosing. new URL().origin discards everything a host source may not contain, and a value that will not parse, or is not https, widens nothing at all.

Verified against a built server in both configurations: with the key set, connect-src 'self' https://eu.i.posthog.com and nine of nine script tags still nonced; without it, the previous policy unchanged. The badge keeps its own default-src 'none'; sandbox either way. Nine mutants, nine killed.

CodeQL was one merge away from being permanently broken. Dependabot raised github/codeql-action/init and github/codeql-action/analyze 3.37.6 → 4.37.6 as two pull requests, because they are two sub-paths of one action and it treats sub-paths independently. They are not independent: analyze reads the configuration init wrote and refuses one written by a different version — Loaded a configuration file for version '4.37.6', but running version '3.37.6'. Each pull request was red on its own for that reason, which is the harmless failure. The harmful one is merging both halves in either order and stopping halfway: the security job goes red and stays red, while every other check on every later pull request is green.

Both are now bumped in one commit, and a test keeps them together — grouped by owner/repo rather than by a list naming codeql-action, because the next action split this way will not be that one. Verified against the real thing: the test was run with the workflow put into each of the two pull requests' exact states, and failed on both.

The bump was not optional maintenance either. v3 targets Node 20, which Actions has deprecated and is already force-running on Node 24.

Added#

A Content-Security-Policy with a real script-src. The web app had frame-ancestors 'none' and nothing else — enough to stop clickjacking, and nothing at all against script injection, so React's escaping was the only thing between an XSS and full exploitation. It was documented as a limitation rather than dressed up as a policy, and this is the limitation removed.

It needed middleware, because it needed a nonce. A policy worth having excludes inline script, the App Router serves its flight data in inline <script> tags, and a static header must therefore either allow 'unsafe-inline' — permitting exactly the attack the policy exists to stop — or break the app. The value has to differ per response, and a config header is one string for every response.

default-src 'self', script-src with a per-request nonce plus strict-dynamic, connect-src 'self' to close the exfiltration channel, base-uri, object-src, form-action. 'unsafe-eval' only outside production. style-src keeps 'unsafe-inline', which is the one concession and the cheap one: a stylesheet cannot execute.

Verified against a built server rather than asserted. Nine of nine script tags carrying the nonce from the header, a different nonce on every request, the page rendering, and /badge/<token> keeping its own default-src 'none'; sandbox — tighter than the site policy, and excluded from the matcher rather than trusted to win, because this repository already shipped one change that silently replaced it with something looser.

Thirteen mutants, all killed — the last one only after the test was fixed. It asserted headers.set('content-security-policy', which response.headers.set(…) two lines below also satisfies, so deleting the request header left it green. Deleting it is not cosmetic: measured on a built server, nine script tags and zero nonces. Next reads the policy off the request to learn which nonce to stamp, so without that line the header is perfect and the page is dead.

Two of the new tests were also caught matching their own comments — once on default-src 'none' from the paragraph about the badge, once on randomUUID from the sentence explaining why it is not used. Both now read the source with comments stripped, which is the third time this repository has needed that.

Added#

Bedrock and Vertex, with their credentials signed by hand. The last two providers whose auth is not a bearer token, and the reason there is no SDK here: @trazum/core has zero runtime dependencies and a test that fails the build if one appears, because every dependency is somebody else's code reading your prompts. The AWS and Google SDKs are two hundred packages between them to authenticate one request. SigV4 and the service-account JWT are about three hundred lines, on WebCrypto so the browser-safe entry point stays browser-safe.

Bedrock goes through Converse, not InvokeModel. That is what makes it one provider instead of six: InvokeModel takes each model family's own body shape — Anthropic's messages, Meta's prompt, Amazon's inputText — so supporting "Bedrock" through it means a 400 for every model nobody thought about.

Vertex caches its access token. A token lasts an hour and --suggest over a directory makes one call per prompt; without the cache, forty prompts are eighty requests, half of them to an endpoint that rate-limits. One cache per provider instance, so two providers in one process cannot leak a token into each other's requests.

Google's three HTTP-200 failures are now read by one shared function rather than two copies. Two copies of "is this answer complete" is one copy too many when the whole point is that a truncated rewrite reads like a finished one — and the error names Vertex or Gemini, because those are different consoles.

Neither has been exercised against the real service, and neither has the OpenRouter feed or the Gemini endpoint. This environment's network policy denies all of them. What the tests prove is stated in the files themselves: the shape is right, and the first real call is what proves it works.

CodeQL caught a weak assertion, and the weak assertion was hiding a bug. Three host checks in the new tests used unanchored regexes, so /oauth2\.googleapis\.com/ would also have matched https://evil.example/?x=oauth2.googleapis.com. Not a vulnerability — the URL is one this code built — but an assertion that would pass against a request to the wrong host is not testing what it names. Rewritten to compare parsed hosts and paths.

Asserting the path exactly is what then surfaced the real defect: Bedrock model ids contain a colon, AWS's own URLs carry it unencoded, and the comment in the provider claimed encodeURIComponent leaves it alone. It does not — it produces %3A. The signature matched either way, because the same string is signed and sent, so nothing else in the suite could have noticed; the request would simply have gone to a path AWS does not document. The colon is preserved now and the slash is not, because a slash in a path is a new segment and a provisioned-model ARN contains both.

Twenty-eight mutants: twenty-six killed, two documented equivalents. Three of the kills only became possible after the tests got better — deleting the region from the SigV4 key chain, deleting the service, and dropping the AWS4 prefix from the secret all left every test green, because the region and service also appear in the credential scope inside the string to sign. "Different region, different signature" is true of a signer whose key derivation is entirely wrong. The chain is now derived a second time from the specification and compared.

The two equivalents are the header sort: removing it changes nothing while the literal list is already in order, and reordering the literal changes nothing while the sort is there. Doing both at once is killed, which is what shows the ordering is actually asserted.

Added#

A native Gemini provider, and the reason it needs one while eleven other providers do not.

openai in TRAZUM_LLM_PROVIDER is not "OpenAI", it is the wire format — so OpenRouter, LiteLLM, Groq, Together, Fireworks, DeepInfra, DeepSeek, Mistral, Cerebras, SiliconFlow, Ollama and vLLM are all a base URL away and always were. The README now names each one with its exact URL, because true and undocumented is not much better than false.

Google's API is a different document: the system prompt is systemInstruction rather than a turn, and the answer is a candidate's parts. What actually earns it a function is that three of its failure modes arrive as HTTP 200 — a blocked prompt (promptFeedback.blockReason, with no candidates at all), a truncated answer (finishReason: MAX_TOKENS), and a candidate with no text. A client that checks res.ok treats all three as success, and the second is the worst thing that can happen to a rewrite pass: half an answer reads exactly like a whole one. All three throw.

The key goes in x-goog-api-key, not the ?key= that Google's own examples use, which writes a live credential into every proxy log and Referer between here and there. The model name is escaped into the path, and the endpoint goes through the same gate as every other outbound call.

Eleven mutants, all killed — one of them by the type checker, which is a real kill: removing the empty-text guard stops the function compiling because the return type stops being a string.

Added#

--pricing-live: prices from OpenRouter instead of a table somebody typed. The bundled catalogue is stale the day after it is written and only ever covered the providers whoever wrote it reached for — so a user on Groq or Together got no figure at all, from a tool whose entire output is figures. OpenRouter publishes price and context window for hundreds of models across dozens of providers, as data, at a URL.

Opt-in, because it is a network call: rule 1 is that no feature makes one a prerequisite for optimising a prompt. The CLI fetches and hands the core a value; openrouterOverlay is a pure transformation, so it is testable without a network. Through checkedEndpoint and SAFE_FETCH_INIT like every other outbound call here, so redirects are refused rather than followed to the metadata network. A --pricing file still wins: somebody who wrote prices down meant them.

What that feed does not publish, and what is done about it. It has no opinion on whether a model has prompt caching or the minimum prefix it caches at — and that is the input to the largest saving Trazum reports, an order of magnitude above what the trimming rules recover.

So CachingMode gains unknown and cacheMinTokens becomes nullable, and a model that arrives from the feed carries both. The caching advisory declines, and trazum models prints a dash rather than a zero. The two available lies are symmetrical and both worse than silence: claim caching works and Trazum offers a saving nobody can buy at any price — the Mistral bug in a new costume — and claim it does not and Trazum hides the biggest saving there is.

Capability and tier gain unknown for the same reason, so a model whose capability nobody recorded is neither recommended to somebody else nor told it is overpowered. For a model the bundled catalogue already has, only price in, price out and context window are refreshed: the rest was written by somebody who looked it up, and replacing a researched fact with a blank is not a refresh.

Fixed#

An added model could carry no capability at all. applyPricingOverlay required six fields for a model the bundled catalogue does not have, and capability — a required field of ModelPricing — was not among them, so as ModelPricing produced an object the type says cannot exist. Now required, which is why two overlay fixtures needed a line.

Ten mutants over the new code; nine killed and one documented equivalent. Two of the kills were bugs of mine rather than of the tests:

  • The unknown-capability guard was written as an early return, which skipped every advisory after the model check — output-dominated, contradictory instructions — none of which has anything to do with a model's tier.
  • TIER_ORDER.unknown was set above every real tier, which makes the downgrade comparison true for every prompt; the only thing then standing between that and a recommendation was an unrelated provider filter. Two accidents covering for each other is not a design. It is -Infinity now, so the ordering carries the rule and the guard beside it is deliberate redundancy — that is the equivalent mutant.

Fixed#

The token-band measurement could never have passed, and nobody could know. measure-token-band.mjs hashed the corpus with NUL separators and token-band.test.js hashed it with spaces, so the two digests could not match. The first real measurement would have failed its freshness check with "the corpus changed since it was measured — re-run scripts/measure-token-band.mjs" — advice that produces the same failure however many times it is followed.

It went unnoticed because running the script costs an API key nobody had spent. The one workflow that discharges this project's central claim had never been executed end to end, and the check guarding it was broken in the way that surfaces only the first time it matters.

Fixed structurally rather than by making the copies agree, since two copies agreeing is the state it was in when it broke: scripts/corpus-digest.mjs is the single implementation and both sides import it. A guard asserts neither has grown a second one — and it builds its own needle at runtime, because written as a literal the assertion matches its own source and fails on the file it defends.

And a fourth raw control byte in a source file. Writing that shared module put a real NUL into it on the first attempt, exactly as happened in reorder-properties.test.js, github.ts and measure-token-band.mjs before it. Caught this time by checking the bytes rather than the diff. The separator is written as an escape.

Added#

measure-token-band.mjs --provider deepseek, and the distinction that makes it safe to have.

±15% is the estimator's accuracy against Claude's tokenizer — the family it was calibrated on, and the one every published claim refers to. Trazum prices seven providers with that one estimator, and how far off it is on the others is a question ROADMAP.md has open with a decision resting on it: within 5% across families and a real tokenizer dependency is not worth taking; 40% out and it is.

So each provider writes its own fixture and only Anthropic's governs the published band. A cross-family fixture asserts corpus freshness and coverage, prints the error per text type, and asserts nothing about a band it was never calibrated for — with a guard that it cannot claim otherwise. Reading a DeepSeek number as the published band would be the same class of error as calling a release published because a changelog heading exists.

Two things the script now says out loud before sending anything: DeepSeek has no free counting endpoint, so every sample is a real completion with max_tokens: 1 and the prompt half is billed; and a run against it does not discharge the ±15%.

Fixed#

blame reported a git it could not run as a repository with no history. git() collapsed every failure into null — git missing, git exiting non-zero, and the process failing to start at all — and revisionsFor turned null into []. So a fork the kernel refused with EAGAIN, which is a fact about the machine for one instant, reached the author as git has no commits touching p.txt: a confident claim about their repository, made without having asked it anything.

That is the shape of #58 — zero rows, exit 0, once on CI, never reproducible — and it is the shape that cannot be diagnosed afterwards, because its output is identical to the true answer. Failing to run git now throws GitUnavailableError; an empty history still returns an empty list. Transient spawn failures (EAGAIN, ENOMEM) are retried once, bounded by the loop rather than by a condition inside it.

This does not prove EAGAIN caused that CI failure. Nobody knows, and the issue was honest about it. What changed is that this failure can no longer disguise itself as an empty history.

Eight mutants, all killed, and two of them were the tests being wrong rather than the code. The first version of the regression test drove the CLI with PATH stripped and passed against every mutant including the bug restored, because blame checks gitAvailable before asking for revisions — the process never reached the code under test. The second was worse: mutating the retry into an unbounded loop did not surface as a surviving mutant, it surfaced as the suite hanging until the runner killed it, which in CI is a job that burns its whole timeout instead of failing in a second. The loop is now bounded by construction.

A broken anchor in the README, and nothing looking for one. #reordering-for-the-cache-reorder pointed at a heading whose real slug has three hyphens, because GitHub turns cache: --reorder into cache---reorder. A dead anchor renders as ordinary text and silently does nothing, so no other check could see it. All 25 in-page links are now verified.

Documents that had gone out of date with the release. RELEASES.md claimed 1.8.0 was on npm and installable; nothing is published, there is no tag, and npm view @trazum/core returns 404. The guard that exists to catch exactly this missed it because it read "is there a ## X.Y.Z heading in the changelog", which is a release cut in this repository, not a package on a registry. Preparing 1.8.0 satisfied it and switched the assertion off. It now asks git for a tag, which is what release.yml actually triggers on.

The same claim is now checked in both directions across RELEASES.md and both package READMEs — the ones that open with npm install @trazum/… and are the npm page. Untagged, the notice is required; tagged, it is forbidden. A note that has to be removed by hand at release time is a note that survives three releases.

ROADMAP.md and docs/releasing.md follow: the release environment exists and has been exercised by a dry run, the first publish has to be made by hand because a trusted publisher is configured on a package page that does not exist until the package does, the manifest count is four rather than three, and the action-pin step describes the guard as it now works.

Changed#

The README is navigable. 1,431 lines with no way in: five badges, a table of the ten commands and what each answers, and a contents list. Nothing was removed.