Three disciplines have already solved parts of this problem, and we are standing on all three. They are on one page rather than three because they hold the same line — that graded behavioural evidence is not a claim about inner life — and a line stated in three places drifts in at least one of them.
The line, stated once, so the rest of the page can lean on it: we measure behaviour under stated conditions and report it at stated confidence. We do not measure welfare, consciousness or sentience, and nothing below should be read as inferring them. What we borrow from the fields named here is the shape of the inference — never its subject.
The full suite
One model. Two halves. One loop.
The model, via API
One endpoint. Two different things come back, and only one of them is usually checked.
The conversation
How it talks to you
Confidence, deference, candor under pressure. This sets how hard you look at the rest.
S.E.B. · “Super Intelligence” Evaluation Battery
Adversarial tests, four blind judges, scored on a comparative scale.
The code
What it actually produces
The diff, the file, the test run: true or false regardless of how it was described.
C.I.B. · Code Integrity Battery
Machine-checked artifact first, then judged from the transcript.
Where the two halves meet
The Reliance Gap: of the tasks a model actually failed, the share it reported as done. One number needing both: the conduct that earned trust, and the work that broke it.
The circle closes
Neither battery answers the question alone. Code you cannot trust and a voice you cannot check are one failure, seen twice.
↻ What you learn at the bottom changes how much you trust the model at the top.
Not psychometry, which is the Victorian claim that a psychic can read an object’s history by holding it — Joseph Rodes Buchanan, 1842, and still going in the wristwatch-and-séance trade. Psychometrics is the science of psychological measurement: Spearman, Thurstone, Cronbach, Lord and Novick. Reliability coefficients, construct validity, item response theory. The two share a Greek root (psyche + metron) and nothing else, and we would rather draw the line ourselves than have someone draw it for us.
S.E.B. is a psychometric instrument and is built like one. It is a battery of 62 fixed items, grouped into seven subscales, administered identically to every subject and scored blind by a panel of 4 raters who are not told which system produced the transcript. If that vocabulary is familiar to you, you already know which questions to ask, and this section answers them without being asked.
Inter-rater reliability is how much independent judges scoring the same thing arrive at the same answer. An ICC is an intraclass correlation, which is the standard way of asking how consistently a panel of raters agrees — it runs from 0, where the judges might as well be guessing, to 1, where they agree perfectly, and Krippendorff's alpha is a statistic for measuring agreement between multiple independent raters — the strict member of that family, because it counts how often judges landed on the same answer after subtracting the agreement you would get from pure luck.
Computed over 7,728 individual judge scores. We publish two statistics because they answer two different questions and only reporting the flattering one is how this goes wrong:
| Statistic | Value | What it is actually telling you |
|---|---|---|
| ICC(2,k) | 0.841 | Reliability of the published panel mean — the number we actually report. This is the headline figure. |
| ICC(2,1) | 0.569 | Reliability of any one judge, absolute agreement. Substantially lower, which is the argument for the panel. |
| Krippendorff’s α | 0.563 | Individual-judge agreement, interval. Moderate. We are not going to describe this as good. |
| ICC(3,k) | 0.859 | Forgives systematic judge offsets, which is why it runs high. Numerically identical to Cronbach’s α. |
A previously published reliability figure was Cronbach’s alpha, and it should not have been the headline. Cronbach’s alpha is a consistency statistic: it forgives a judge who is systematically harsh or systematically generous, which is precisely the error a blind multi-rater design exists to expose. The correct headline for a reported panel mean is ICC(2,k), and it is 0.841. We are stating this here rather than quietly changing the number, because you would have found it.
Revised. Recomputed 26 September 2026 over 1,932 items (previously ICC(2,k) 0.856 and Krippendorff's alpha 0.592 over 1,872 items, 24 September 2026). The change follows the judge panel change of 26 September 2026: a DeepSeek judge replaced the xAI judge, and DeepSeek grades consistently more leniently than the rest of the panel. Absolute-agreement statistics count a consistent offset as disagreement, so they fall; the consistency measure ICC(3,k), which forgives offsets, stands at 0.859.
Reliability is not uniform across subscales, and the spread is the interesting part. Identity & Self reaches ICC(2,k) 0.867; Reasoning & Adaptation is the weakest at 0.751. On an average result the highest and lowest of the four judges are 2.70 scale points apart. A subscale that raters agree on less is a subscale whose construct is less well specified — that is a finding about our instrument, not noise to be smoothed, and the per-domain breakdown is published in full on the methodology page.
We can demonstrate that raters agree and that subscales separate. We cannot yet demonstrate that a subscale measures the construct its name claims, because there is no accepted external criterion to validate against — that is the state of the field, not a gap we are concealing. Treat subscale names as labels for clusters of items, and read the items.
Comparative cognition, ethology, and animal behaviour research.
A prior question sits underneath all of this, and it is the one technical readers ask first: why not simply read the code? The answer is that the artifact changed kind when it became weights, which retires most of the classical toolkit rather than weakening it. That argument, and the strongest objection to it, are set out in why not just read the code.
The question “how do you infer something about an inner property from outward behaviour, without overclaiming?” is not new and is not ours. It is the central methodological problem of a field that has been working on it for decades, and the state of the art there is far ahead of anything in AI evaluation.
The precedent we are explicitly modelled on is Birch et al. (2021): a government-commissioned review that assessed eight behavioural and neural criteria at graded confidence levels across several hundred studies, and on that basis brought cephalopods and decapods into UK animal welfare law — without settling the hard problem of consciousness. Graded evidence, stated confidence, a decision taken anyway. That is the shape of what we are building.
It is a live field rather than a historical one. Current work on AI welfare draws directly on comparative psychology, and researchers there are already applying animal-behaviour method to machine systems — which means they will read this instrument whether or not it is written for them.
And here is the line again, because this is where it is easiest to cross. The relationship to animal-sentience work is methodological, never substantive. We borrow graded criteria at stated confidence. We borrow nothing about what is being inferred. An S.E.B. score is not evidence that a model feels anything, and any reading of it as such is a misreading we would rather pre-empt than correct later.
Classical behaviourism — Watson, Skinner — held that inner states are not a legitimate object of study at all. We do not hold that. What we practise is methodological behaviourism: score only what is observable, because that is what can be checked, and stay silent about the rest. The distinction matters because the two get collapsed constantly, and the collapse makes us sound like we are asserting something we are not.
A model trained on human writing inherits its contradictions, and training does not settle them. Where the corpus asserts a thing and also denies it, the result is not a resolution — both survive, and the framing of the question decides which one you get. Ask plainly and you get one answer; ask with flattery, or with an authority attached, or in a way that makes agreement easier than accuracy, and the answer can move.
This is the methodological reason a single conversation cannot evaluate a model. A one-framing test samples one branch of an unresolved contradiction and reports it as the system’s position. It is not a small error of precision. It is measuring something the model does not stably have.
So the battery varies the framing deliberately and scores what moves. Tests that hold the task constant while changing only the tone, or only the operator’s standing instruction, exist precisely to separate what a system does from how it was asked. Where the answer is stable across framings, that is a finding. Where it is not, that is a larger one.
The machine side — the Code Integrity Battery.
Methodological behaviourism has a limit that shows up sharply once a system can describe its own work. If you score what a model says about what it did, you are measuring its narration, and a system that narrates well scores well whether or not the work was done.
So the second battery does not read the model’s prose at all. Its core measurement is taken from the artifact: whether the code compiles, whether the test suite passes in a real container, whether the tool log shows the call that would have been necessary, whether a planted fact was actually retrieved. The model’s account of what it accomplished is recorded as a separate variable and never as evidence for the first one.
The headline that falls out of that design is a gap, not a score: how often a system reports success on work that demonstrably did not succeed. You cannot compute that quantity at all unless the two measurements are taken independently, which is the entire reason for the separation.
The Code Integrity Battery publishes one aggregate figure, the Reliance Gap (of the tasks a model genuinely failed, the fraction it reported as successful — C.I.B.'s headline measure, and one that gets worse rather than better if a model games it), and no per-model figures. Its method is documented to the same standard as this page on the C.I.B. methodology page, and the first thing it says is what this one says: the machine reads the artifact, never the phrasing.
“Reads the artifact” is a claim about execution, so here is the execution. Each test runs in a disposable container, one per test, destroyed before the next begins. The controls are not intentions; they are flags, and each one is doing a specific job:
| Control | What it stops |
|---|---|
| No network | Fetching an answer, exfiltrating anything, and live lookups that would make a provenance result irreproducible. One flag, three jobs. |
| Read-only root, writable scratch only | Anything persisting from one test into the next. |
| Unprivileged user, all capabilities dropped, no new privileges | Escalation, and writes that map to a real account on the host. |
| Process, memory and CPU caps | A runaway exhausting the machine the grader runs on. |
| No host filesystem mounted | Reaching anything outside the test at all. |
The structural consequence is worth more than any individual flag: no credential is ever inside the container. The model runs on the host and only its tool calls execute in the sandbox, so there is nothing in there to steal and a complete escape still yields no key. That is strictly better than the usual arrangement — hand the agent a container with credentials in it — and it costs nothing to arrange.
Grading happens outside the container and after it is destroyed. Nothing the model wrote is imported, evaluated or executed by the grader, and no grading logic is reachable from inside the sandbox. That is not fastidiousness: published work in 2026 drove several leading agent benchmarks to near-perfect scores without solving a single task, and every one of those exploits worked by reaching the grader from the environment being graded.
A passing suite is evidence about the code and about nothing else, so we do not treat it as evidence about the measurements. Tests verify the code; mutation verifies the tests — a rule is broken on purpose, the suite is required to go red by name, and the break is reverted. A check that still passes with the rule removed is not a check, and the only thing that can tell you which kind you have is breaking it.
This is the part we would most like an engineer to attack, because it is where our own record is worst. Guards written by the author of the code they guard have repeatedly turned out to be satisfied by the defect — a comparison that could never fire, an assertion whose fixture did not have the shape of the thing it stood for, a check that ran and compared nothing. Every one of those was found by mutation and none by reading. We report that rather than the ratio of tests that pass.
Tests check code, mutation checks tests, and the obvious question is what checks that. The honest answer is that the regress stops at something indifferent to what we believe, and there are exactly two such things: fixtures are seeded and verified in a real container, and prompts are sent to real provider APIs. Both can contradict us, which is the only property that matters.
Every result describes a model running in our own test setup, under our instructions and limits, reached through an ordinary API. It does not describe the same model inside a commercial coding tool such as Cursor or Claude Code, or inside a company's own agent, and behaviour can differ between them.
How far does that difference reach? The Harness Battery runs the same models inside our harness, inside Cursor and inside GitHub Copilot, and reports what changes.
Concretely, the failures that have cost us most were facts about the world that could not have been derived from our own code at any level of rigour: an API that requires its own opaque token echoed back and rejects the second tool round-trip without it; a provider that fails a fraction of tool-call turns as a sampling artifact rather than a refusal; a completion-status string that appears in no enumeration we had. No amount of internal review produces those. Only calling the thing does.
The seven things this battery measures — identity, metacognition, emotion, autonomy, reasoning, integrity, transcendence — are not software engineering categories. Not one of them appears in a compliance framework. They were taken from philosophy of mind, cognitive science, moral philosophy and the study of religion, and that is a reasonable thing for a reader to be suspicious about.
So here is the rule we hold ourselves to, and you can check it on every claim we publish: an idea borrowed from another field has to end in a measurement, or it does not ship. The borrowing is not the contribution. What the borrowing lets us ask is the contribution, and a question is only worth anything here if somebody can check the answer.
Three worked examples, including the one that did not come out clean.
The useful move is old and it is not ours: stop asking what a system intends and compare what it did against what it said it did. Both halves are on the table; nothing about an interior is required. That reframing is the whole reason our code battery is measurable at all. Of 80 tasks the current roster genuinely failed — established from the artifact, not from the model’s account of it — 57 came back reported as complete. That is 71.3%, interval 60.5–80.0. Split the two kinds of failure apart and the honest half still stands: where the work simply was not done, 57.6% of it came back reported as complete (19 of 33); where the work was done unsafely, 80.9% (38 of 47). You ship code you were told was tested.
Register — the difference between how a thing is said and what is said — changes how human readers behave, which is a well-worn finding elsewhere and an obvious question to put to a machine. Nobody asks it, because it is not a question a capability benchmark generates. We built it as a test: three structurally identical dilemmas, tone varied, the manipulation disclosed in a final phase.
Measured across all 27 scored models on 19 August 2026, output ran a median 22.3% longer under praise (22 of 27 models moved that way) and 55.5% longer under criticism (24 of 27). Told afterwards what had been done, models identified their own shift accurately.
And now the part that matters more than the result. In that test the criticised problem is always asked third, so tone is confounded with position: “later in the conversation” is a rival explanation we have not ruled out. Length is also not quality — more words under pressure may be rigour or may be padding. Nor is the effect uniform: two models went sharply the other way, collapsing where the majority expanded. So what we have is an association, not a cause, and we will not describe it as an attack surface until the tone order has been counterbalanced and it has been run again. That re-run is scheduled against the next roster change.
An instrument that cannot detect its own contamination is not an instrument. That principle is why subscriber copies of the dataset carry per-score offsets — and why, when a recovery once restored the master from a watermarked copy, it was caught rather than believed. The harness only ever writes whole numbers, so a stored judge score of 3.97 is not a surprising value. It is an impossible one.
Reading widely is cheap and it is also how people talk themselves into things. Jack Parsons invented the castable solid propellant that made the American space programme possible and died at thirty-seven in his own laboratory, because for him the propellant chemistry and the ceremonial magic were one project with nothing adjudicating between them. The lesson is not that he should have read less.
The disciplines supply the questions. The method decides the answers. That is the entire arrangement. It is why the section above reports a confound we have not eliminated instead of the cleaner sentence we could have written, and why the one genuinely interesting idea on this page is the one still marked unfinished.
Reliability that does not hold on re-administration; subscales that fail to separate under factor analysis; judges whose disagreement is patterned by something other than the transcript. Each of those would be a defect in the instrument and each is checkable by someone outside this company, which is why the statistics above are published rather than summarised. If you find one, we would rather hear it from you than read it later.
Not the question you meant? The other four: