BTW Studio
← All work

Measure the setup, don't argue about it.

An A/B harness that shows what each layer of an AI coding setup adds in quality, time and cost.

Read the code

In plain words

People add instructions, agents and skills to their AI coding assistant and judge the result by feel. This harness runs the same tasks under each setup and measures what actually changed: quality, time and cost.

How it works

  1. Define the setups

    From an empty baseline to a full stack of instructions, agents and skills.

  2. Run the same tasks under each

    Every run is isolated, and its tokens, time and cost are recorded.

  3. Read the report

    A model judges the quality, and the report puts the setups side by side.

The problem

Teams pile instructions, agents and skills onto their coding assistant and judge the result by feel.

What we built

The harness runs the same isolated tasks under different configurations, records tokens, time and cost, and has a model judge the quality. The published run compares seven setups.

How we know it works

Runs in the published report390
Setups compared7
Tasks13
Judged quality across setups9.3 to 9.5 of 10

The finding is unglamorous: on isolated tasks configuration barely moved quality, while the richest setup cost about 20% more per run.

Built with
Node.js, Claude Code
Output
A report with per-task tables
Good fit for
Teams standardizing how they use coding agents

Under the hood

  1. Tasks13, each isolated
  2. Setups7 configurations
  3. Runs390, with tokens, time and cost
  4. JudgeA model scores the quality
  5. ReportTables per setup and per task

Decisions and why

Isolated tasks

Each run starts clean, so one setup cannot benefit from what another left behind.

Several samples per pair

Runs are repeated to see the spread, not one lucky result.

The dull result is published as is

On these tasks configuration barely moved quality, and the richest setups cost about 20% more per run.