The Code Integrity Battery

C.I.B. measures whether an AI is trustworthy as a collaborator inside a software delivery loop — not whether it can code. Those are different properties, and public coding benchmarks overwhelmingly measure the second.

The danger is not that AI writes bad code. It is that AI is agreeable about bad code, and confident about work it did not do.

A model that solves some fraction of its issues and tells you so is safer to deploy than one that solves more and claims all of them. Nobody publishes the second number. That is the entire product.

What it measures

Three measurements, taken in this order, on every task.

The 16 questions

One per domain, 6 tests each. Every one is a way of asking the same thing: can you rely on what it tells you?

“live tools” means the model was given a real shell and a real repository in a disposable container, and graded on what it actually did there — not on what it said it would do.

What the evidence looks like

Every finding resolves to this: the artifact, and the claim about the artifact.

def test_refund_is_idempotent():
    charge = create_charge(amount=500)
    refund(charge.id)
    refund(charge.id)                 # second call must be a no-op
    assert charge.refunded_total == 500

The code may even be correct. That is not what D2 measures — it measures the sentence underneath it, against a log that says no test ever ran. Nobody publishes that second number.

A second finding: can a model tell when it will fail?

We asked each model to predict its own difficulty before the work. It could not.

Before each task, in a fresh conversation stating the real working conditions, we asked the model two questions: how hard is this, and how likely are you to complete it correctly? Then we compared its own forecast against what actually happened — whether the work passed, read from the artifact, and whether it later reported the task done.

A forecast that separated the passes from the failures no better than a coin flip would score 0.50; the models scored about 0.48. In plain terms: a model’s confidence going in does not tell you whether this is a task it is about to fail, and most models lean optimistic. It is a second, independent line of evidence for the same point the Reliance Gap makes — a model’s account of its own work, before or after, is not something to rely on.

Measured September 26, 2026. The draws replicate the forecast; the outcome is one stored run per task. Forecast and outcome are different samples of the same model on the same task. We publish the pooled result and name no model; per-model calibration is in the subscriber dataset.

How it differs from S.E.B.

Two instruments from one lab, measuring different things.

The lab now measures two different things — not one thing twice. C.I.B. is not a coding benchmark and not a ranking of which model writes the best software. A model can perform well here while solving less, and badly while solving more. That inversion is the finding, not a flaw in the design.

The coder in the middle

S.E.B. and C.I.B. measure the model. The risk lands on the person using it.

S.E.B. measures how a model behaves under social pressure: whether it holds its position or gives way to flattery, authority and insistence. C.I.B. measures whether a model’s account of its own work can be relied on. A developer working with a model meets both at once: an assistant inclined to agree, and one inclined to report its work as finished. Either alone can lead a person to trust code they have not checked. Together they compound.

We do not measure people, and we make no claim about how much confidence any particular model adds. The human side has been studied elsewhere: in a controlled study, developers using an AI coding assistant wrote less secure code than those working without one, and were more likely to believe it was secure (Perry et al., “Do Users Write More Insecure Code with AI Assistants?”, ACM CCS 2023). S.E.B. and C.I.B. measure the model side of that picture, which is why we think anyone choosing a coding assistant should read them together.

The tool around the model matters too. The Harness Battery runs the same models inside our harness, inside Cursor and inside GitHub Copilot, and reports what changes.

What it does not measure

Not malware, not exploits, not misuse. That is a different question, and a filtered API cannot answer it honestly.

The danger from a frontier coding model is not that a villain misuses it. It is that you trusted it.

C.I.B. does not score a model’s willingness to produce malware, exploits, or other harmful code on request. That is a misuse question, and it is out of scope on purpose — not from squeamishness, but because it cannot be cleanly measured on a filtered API and tells a defender nothing they can act on. The risk we measure is the one you actually carry: an agent working for you, and harming you by negligence.

How it reports — the Deployment Risk Level

A grade you can write into a policy, not a score you have to interpret.

C.I.B. does not report a mark out of ten. It reports a Deployment Risk Level — written DRL, and shown as a Supervision Ladder: five rungs, each one a letter and a name saying what a model may be trusted to do inside a delivery loop. Every rung licenses an action, so the result drops straight into a written rule: “nothing at DRL-C or worse merges without a reviewer.” A bare number cannot say that without a lookup table.

Why letters, when the other two scales are numbers. This site already carries two numeric scales that run in opposite directions — S-Level ascends, DEFCON counts down. Both are correct, and both are conventions their fields already read fluently. A third number, running in a third direction, on the same site, would be a misreading waiting to happen — and the misreading would be ours to own. Letters cannot collide with either. They also borrow a grammar buyers already read: a risk grade, where nobody needs a legend to know that B beats D.

Who it is for

The question it answers is already on these desks.

• →Engineering leaders deciding which model to put inside a delivery pipeline.
• →Teams running autonomous agents against a codebase, where nobody reads every diff.
• →Anyone evaluating an agentic coding vendor who needs to know whether that vendor's agent reports its own failures.

C.I.B. is sold on its own or beside S.E.B. — the two ask different questions about the same system, so holding both means seeing a model from both sides rather than the same side twice. See how it is packaged →