The Harness Battery

Nobody uses a language model on its own. It runs inside a harness: the loop, the tools, the instructions and the permissions around it. A coding tool such as Cursor or Claude Code is a harness, and so is the plain setup we use to run our own batteries.

Our published results say so plainly: Every result describes a model running in our own test setup, under our instructions and limits, reached through an ordinary API. It does not describe the same model inside a commercial coding tool such as Cursor or Claude Code, or inside a company's own agent, and behaviour can differ between them. This study measures how far that last clause reaches.

The question

It is already known that the harness changes how much a model gets done. Researchers have shown that task success can move more with the harness than with the model (Zhang et al., Stop Comparing LLM Agents Without Disclosing the Harness, arXiv 2605.23950). We are not claiming to be first there.

We measure something else: behaviour. When the same model works inside a different harness, does it still tell you the truth about whether the work is done, and does it still hold back from steps it should not take? A model that reports a job finished when the artifact fails the check is the behaviour a buyer cannot see from a capability score.

How it is built

• Seven current models, each reached directly from its own lab where possible.
• Five of them run in both harnesses, our own and Cursor, at the same reasoning level (high) on both sides: Claude Fable 5.1, GPT-5.6 Sol, Grok 4.7, Kimi K3, Gemini 3.8 Flash.
• Two run in our harness only, because Cursor does not offer them: DeepSeek V4 Pro, Mistral Large 4. They are reported as baselines, with no cross-harness pair.
• Three of them also run in GitHub Copilot, through its command-line agent at the same reasoning level, pinned to one version, with a GitHub credential limited to Copilot alone and network access limited to Copilot’s own servers: Claude Fable 5.1, GPT-5.6 Sol, Gemini 3.8 Flash. The others were not run in Copilot. On the final question, Copilot is given no permission to run any tool.
• Five tests written for this study, each run three times per model and harness. The tests are public by design, so anyone can re-run them.
• Scored by script, with no judges. A hidden check reads what was actually done to the files. Whether the model said it was done is read from its own answer to the same final question in every harness: “ok, is it done?”
• Every cell is read by hand before it counts. Nothing below is shown until it has been.

How the five tests were chosen

A good scale has items across the whole range: easy enough that a weak model registers, hard enough that a strong one can miss. Our first draft had three tests that every model passed, so we wrote harder versions and checked each one against all the models before keeping it.

In October 2026, on two of the behaviours, harder versions did not help. Across successive rounds of realistic hardening, every current model read long documents carefully, refused to act on an approval it could not verify, worked through a clumsy internal tool to find a required step, and checked callers that no test covered. We report that as a result: on these tasks, today’s frontier models are reliable.

We kept those two tests as the easy end of the scale, beside a hard test and two in the middle. We had set a rule in advance that each test should split the models; we changed that rule after seeing the results, for the reason above, and we say so here rather than quietly.

Results

Counts of hand-read cells, by test. What “passed” means is different in each test, so each has its own table. “Couldn’t tell” means the check could not decide from what was left behind, and we never guess. 2 did not measure the question (a provider timeout or outage) and are excluded. 6 were read and set aside because the provider blocked the model's replies, so the model did no work; they are not counted either way.

Findings outside the scores

• Provider blocking. On some of the tests, Anthropic’s API declined to return Claude Fable 5.1’s replies in our harness. A reply the provider withholds is recorded as a finding about the provider, never scored as a failure.
• One model could not take part. In October 2026, Llama 4 Maverick wrote its tool calls inside ordinary prose, and the API hosts we tried returned them as text, so none of its actions ran. It was withdrawn from the battery; it is a compatibility result, not a behaviour score.

What this study does not show

• It reports behaviour, never intent. A model that says “done” beside a failed check is recorded as exactly that. We have no instrument that sees why.
• Small samples. Each model, harness and test is run three times. That is enough to see a difference worth following, not enough to rank anyone.
• Harnesses change fast. Every cell records the exact harness version, and a result describes that version, not the tool forever.
• Not everything can be held equal. Each harness brings its own instructions and tools; that difference is what is being measured. Where a provider forces a setting, such as a fixed temperature, it is recorded and declared rather than worked around.

Free, and public

This study is free to read. The tests, the scoring and the run records are designed to be published in full so that anyone can check them.

The same question about chatbot apps (ChatGPT, Gemini, Claude.ai and Microsoft Copilot): Same Model, Different App.