Why a pass flag is a number, not an answer
A green test suite tells you that nothing it checked went wrong. It does not tell you how much it checked, and those are different facts. The gap between them is where most of the risk in AI-generated code actually sits.
None
Nothing was demonstrated. The candidate did not get far enough to show anything.
Syntax
It parses and builds. Nothing has been executed.
Runtime
It runs. Whether it does the task is still open.
Partial behavior
Some required behaviours are satisfied, and the record says which.
Complete behavior
The required behaviours are satisfied, with coverage recorded separately.
A pass flag collapses all five of these into one value.
What a candidate demonstrated, recorded on an ordered scale rather than as a flag.
Binary results hide their own coverage
Two changes can both come back green while one exercised the behaviour that mattered and the other never touched it. The flag is identical. The evidence behind it is not. Once those two changes are sitting in the same branch, nothing distinguishes them, and the safe assumption becomes that neither was checked.
That assumption is expensive. It means review effort spreads evenly across work that deserves wildly different amounts of it.
What a graded scale records instead
Inferstep records what each candidate demonstrated on an ordered scale, from none through syntax and runtime to partial and complete behavioral evidence. Coverage of the required behaviors is recorded separately from the grade, because how thoroughly something was exercised is a different question from whether it passed.
- A candidate that fails to parse demonstrates nothing.
- A candidate that builds demonstrates syntax and no more.
- A candidate that runs demonstrates runtime behaviour.
- A candidate that satisfies the required behaviours demonstrates those behaviours, and the record says which.
Three ways a candidate is rejected
Grading only means something if the obviously wrong answers never reach the scale. Detected execution failures are rejected. So are candidates that execute without implementing the task, because code that runs while doing nothing useful survives review more often than a visible failure does. So are candidates whose cross-file calls resolve to no in-scope definition.
When nothing survives, the runtime repairs against the recorded failures rather than starting over, which is the difference between using a failure and discarding it.
Provenance is the other half
A generated alternative has to earn authorization before it displaces what your agent proposed. When none earns it, the proposed content can still be written after separate syntax and structural checks, and it carries no verification metadata. It is never reported as checked.