Measure the setup, don't argue about it.
An A/B harness that shows what each layer of an AI coding setup adds in quality, time and cost.
In plain words
People add instructions, agents and skills to their AI coding assistant and judge the result by feel. This harness runs the same tasks under each setup and measures what actually changed: quality, time and cost.
How it works
Define the setups
From an empty baseline to a full stack of instructions, agents and skills.
Run the same tasks under each
Every run is isolated, and its tokens, time and cost are recorded.
Read the report
A model judges the quality, and the report puts the setups side by side.
The problem
Teams pile instructions, agents and skills onto their coding assistant and judge the result by feel.
What we built
The harness runs the same isolated tasks under different configurations, records tokens, time and cost, and has a model judge the quality. The published run compares seven setups.
How we know it works
| Runs in the published report | 390 |
|---|---|
| Setups compared | 7 |
| Tasks | 13 |
| Judged quality across setups | 9.3 to 9.5 of 10 |
The finding is unglamorous: on isolated tasks configuration barely moved quality, while the richest setup cost about 20% more per run.
- Built with
- Node.js, Claude Code
- Output
- A report with per-task tables
- Good fit for
- Teams standardizing how they use coding agents
Under the hood
- Tasks13, each isolated
- Setups7 configurations
- Runs390, with tokens, time and cost
- JudgeA model scores the quality
- ReportTables per setup and per task
Decisions and why
Isolated tasks
Each run starts clean, so one setup cannot benefit from what another left behind.
Several samples per pair
Runs are repeated to see the spread, not one lucky result.
The dull result is published as is
On these tasks configuration barely moved quality, and the richest setups cost about 20% more per run.