01
One fixed task
An 11-question audit of a purpose-built Python service: route count, auth entry point, schema, a planted N+1 query, one hardcoded secret, and more. Every answer was machine-checkable.
BENCHMARK · AUDIT-V1 · N=10 EACH
Same model. Same answer. Half the bill.
Ten runs each of an identical, single-prompt code audit. Same prompt, same effort, same Opus 4.8 model. Both graded runs scored 11/11. Manna Code averaged $0.43 (Opus 4.8, xhigh); Claude Code averaged $0.88 (Opus 4.8, xhigh).
AVERAGE COST PER RUN
Manna Code$0.43
Claude Code$0.88
2.06×
cheaper per average run
COSTS NORMALIZED TO PUBLISHED API RATES
MANNA CODE / RUN
$0.43
average · n=10
CLAUDE CODE / RUN
$0.88
average · n=10
COST REDUCTION
51%
2.06× cheaper
QUALITY
11/11
one graded run each
METHOD
Measure the cost to finish the same job.
The experiment fixes the task, prompt, model, and effort. Real token usage determines the average cost; correctness was individually checked on one run per tool.
02
One fixed model
Identical prompt, same model and effort—Claude Opus 4.8 at xhigh—a fresh session per run, and read-only access. The only variable was the harness around the model.
03
Correctness check
One run per tool was individually graded, and both scored 11/11. The remaining 18 runs were inferred complete from consistent output profiles.
04
List-price costing
Fresh input, cache reads, cache writes, and output tokens were priced at published API rates. Every run reconciled to the cent against the tool's own accounting.
EVIDENCE
The result — aggregate, and every individual run.
Average cost per run
Ten runs per tool; bars are to scale.
Manna Code$0.43
Claude Code$0.88
Average tokens per run
Total tokens; bars are to scale.
Manna Code149K
Claude Code439K
Cost per run — every run, both tools
All twenty runs, individually. Vertical marks are each tool's average.
WHY NOW
Metered models expose the re-reading tax.
The Fable 5 move to metered usage makes the mechanism newly urgent: every repeated context token becomes a visible line item.
Audit-v1 measured Opus 4.8, not Fable 5. Its demonstrated result is deliberately narrower: Manna Code moved about 2.9× fewer tokens on the same benchmark task, while both individually graded runs scored 11/11.
Keep the models you value. Make the harness move less waste.
HONEST LIMITS
What this benchmark does not prove.
- One task, one model, one single-prompt scenario, n=10 per tool. This is not a broad benchmark suite.
- One run per tool was individually graded. Both scored 11/11; the remaining 18 runs were inferred complete from consistent output profiles.
- Cost uses published API list prices, not the effective price of any subscription plan.
- The test says nothing yet about multi-turn economics. A dedicated long-thread benchmark is separate work.
- No Codex or GPT comparison was run. This page reports only the same-model harness comparison.
LOCAL-FIRST · BYOK · NO COMPACTION
The model did the same work. The harness moved fewer tokens.
Put Manna Code on your machine, bring the model keys you already use, and keep every session's cost visible.
TASK AUDIT-V1 · 2026-07-07 · CLAUDE-OPUS-4-8 @ XHIGH · N=10 EACH · EVERY RUN RECONCILED TO THE CENT AGAINST EACH TOOL'S OWN ACCOUNTING