Skip to content
Trazum

1.60.0

Our own medicine, measured

A minor closes an arc, and this one closes the last arc the 1.52–1.60 plan named. It does not close the plan: 1.54.0 and 1.57.0 are blocked on provider credentials this repository does not have, and 1.58.0 is an editor extension — a distribution commitment rather than a feature. Six of the nine arcs are delivered; the other three stay open and named, which is what the plan said it would do with an arc it could not build rather than renumbering the gap away.

The arc committed in advance to a scoreboard, and it reports one of three. The record is self-reported is no longer true. No outcome is recorded is weakened, not overturned. This project has no usage log of its own stands in full, with the reason written down rather than worked around.

Everything below is about this project rather than about the product, except ignore, which had to exist before the project could take its own gate.

Added#

And the arc closes on the number it could not produce, which is what it said in advance it would do. The fourth and final chapter of arc 1.60.

AdmissionAfter the arc
The record is self-reportedNo longer true — five defects found by CodeQL and by nothing here
No outcome is recorded for any of itWeakened — one outcome on the record, the 0.5% above
This project has no usage log of its ownStill true, in full

Why the third stands: this project would have to spend money on models and record it, and it does not spend. What could be counted is the cost it imposes, which is counted, under a heading that says it is a different sentence.

Why the second is only weakened: whether it helps needs somebody it helped. Whether it is used has one available instrument and it was refused on the record — npm download counts are fetches, not uses, and mirrors, CI runners and bots are in the total, so the figure bounds above and nothing bounds below. A floor can prove "over" and can never prove "under" is on this project's own doctrine list; quoting a ceiling as evidence of adoption is that rule inverted.

One of three is less than the arc hoped for and exactly what it committed to report. A guard grades the scoreboard against the page, so a later edit cannot promote a second admission quietly.

ignore — saying which files are prompts, when the extension cannot. A new config key, and the feature that had to exist before this repository could take its own gate. Directory mode decided from the extension alone, so a repository with a corpus of .txt files got every fixture walked, budgeted and baselined, and there was no way to say otherwise — no ignore list, no include list, nothing.

{ "extensions": [".txt"], "ignore": ["**/fixtures/**", "**/corpus/**"] }

Globs relative to the walk root. A matched directory is not descended into at all, so an ignored tree costs one comparison rather than one per file in it. A pattern that climbs out of the project with .., or an absolute one, is refused the same way a budget pattern is — on a pull request the config comes from whoever opened it. Threaded through every walk that already consulted extensions, because the two halves of what counts as a prompt here disagreeing is the same defect one layer down.

Found by needing it: the first honest config for this repository made trazum doctor report 35 of 37 prompts have no budget and list this project's own test fixtures.

And the loop this product sells was inert in the repository that sells it. The third chapter of arc 1.60. trazum init writes a config, trazum baseline records what a repository's prompts cost, trazum check fails a build when they grow — all three shipped arcs ago, all three are what docs/ci.md tells other people to run, and this repository had no config and no baseline of its own.

trazum.config.json and trazum.baseline.json are committed now, CI runs the gate, and the gate was proved by growing the prompts past the limit and watching it exit 1 rather than by reading the flag.

A gate flag that silently gates nothing, found the first time the CLI was pointed at this repository: check --baseline against a config with no baseline block prints nothing and exits 0, because the flag is read as config.baseline !== undefined && boolFlag(...) — a missing block disables the gate instead of failing the run. A green build, from a command invoked with the flag that asks for the check.

And the reason no baseline was ever committed here. At the root with the default extensions, trazum baseline records 74 prompts and 509,255 tokens — the README, the changelog, the roadmap — because directory mode defaults to .txt .md .prompt .tmpl and this is a documentation-heavy repository. The gate is scoped to examples/, and the scoping is the honest fix rather than a workaround: the prompts this project actually ships to models live inside .ts, where the baseline gate cannot see them at all.

Guards: the committed baseline is checked against the tree file by file, the config's baseline block is asserted present (without it the gate is the no-op above), the workflow is asserted to still carry the step, and the gate itself is run against a scratch copy grown past the limit.

This project had never counted the tokens it puts on your bill. The second chapter of arc 1.60. Four system prompts ship inside @trazum/core and are sent to a model on every --llm, --suggest, --semantic and examples-review run — on your key, on your bill, before a single token of your own prompt is counted. A tool that reports other people's prompt cost and had never counted its own is the self-report problem in its most literal form.

Measured by running the optimiser on them, at the aggressive level:

PromptTokensRecovered
suggest2912
semantic3824
refiner1980
example-review3050

1176 tokens, and this project's own rules recover 6 of them — half a per cent. Eleven of the twelve rules are inert on all four. The honest reading is not that the rules are bad: these are prompts written to be read by a model, with no politeness, no hedging and nothing repeated, which is the shape the dictionaries have nothing to say about. The same result rules --measure produced on examples/.

What it establishes, and what it does not, both on our own medicine. It records one outcome — on the only corpus this project owns, the feature this product leads with recovers 0.5% — which by the standard set in 1.50.4 is a real outcome rather than an inference, and the first on that page about the product working rather than about the process around it. It establishes nothing about whether users benefit (four prompts written by the person who wrote the rules are the least representative corpus imaginable), nothing about the model-side passes those tokens buy, and it is still self-reported — the optimiser marking its own work, one layer in.

The admission it does not overturn. This project has no usage log of its own is still true and the page still says so: this measures the cost the project imposes, not the cost it incurs. A guard asserts the page keeps saying so, because claiming an admission fell when it did not would be the exact failure the page exists to catch.

The uncomfortable arithmetic is published rather than left to a sceptic. At about one per cent recovered, any prompt under roughly thirty thousand tokens costs more in this project's own instructions on a single --suggest run than the rules recover from it. Different budgets — a per-call cost you opt into against a saving on every call forever — and the first thing a sceptical reader would compute.

The guard derives the four prompts from the package's own exports, so a fifth shipped without being measured fails the build; the published figures are checked against what the optimiser produces, so they cannot drift from the prompts; and both are proved against a fabricated fifth export and a fabricated row.