Inferstep

5 october 2026 · explainer · inferstep lab

What a passing test proves about AI-generated code

A check answers a narrower question than it seems to. What parsing, types, builds, sandboxes and tests can and cannot tell you, how ATLAS layers its own checks, and four faults we found in them.

A park by a river, painted in small dots of colour.
Georges Seurat, A Sunday on La Grande Jatte — 1884 (1884–86), Art Institute of Chicago. Public domain.

A program can compile and still be wrong. It can pass its tests and still be wrong. Most of verification comes down to knowing which question a given check actually answered. That is hard enough when a person writes the code. It gets harder when a model does, because the model often writes the tests as well.

the oldest lesson

In October 1969, at a NATO conference on software engineering in Rome, Edsger Dijkstra said: "Testing shows the presence, not the absence of bugs." His own notes from the same year read: "Program testing can be used to show the presence of bugs, but never to show their absence!"

His argument was arithmetic. A multiplier for two 27-bit numbers has 254 possible pairs of inputs. Testing them all, he calculated, "would add up to more than 10 000 years", and so "exhaustive testing, even of a single component such as a multiplier, is entirely out of the question." Every test suite is a sample.

the oracle problem

A test has two parts: an input, and a way to decide whether the output is right. The second part is called the oracle, a term introduced by William Howden in 1978. Barr and colleagues, in the standard survey of the subject (2015), define the problem it names: given an input, the challenge of "distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior."

Generating inputs is the part machines do well. Deciding what the right output is remains difficult, and where automation falls short, the survey notes, "the final source of test oracle information remains the human."

Some oracles come for free: a crash, a timeout or a segmentation fault is almost always wrong, whatever the program was meant to do. The survey calls these implicit oracles. They are useful but weak, since a program can finish cleanly and print the wrong answer.

Three overlapping dotted regions: what the user needs, what the tests check, and what the code does. Only the small region where all three meet is densely dotted.
The oracle gap. On the left, what the user needs; upper right, what the tests check; below, what the code does. They overlap only in part, and code can pass every test while falling outside what was needed.

a ladder of checks

Checks can be ordered like rungs on a ladder. Each rung rules something out and leaves something open.

parses well-formed text; nothing else is known names and types whole classes of error ruled out builds the toolchain accepts it; no behaviour runs without crashing on the paths that actually ran passes tests on the inputs tried, by their oracle holds its properties as strong as the property, no more does what was asked needs the request, and often a person coverage and mutation measure the tests themselves
A ladder of checks. Each rung is cheaper than the one above it, and each answers a narrower question than it may seem to.

Parsing is the bottom rung: Python's py_compile, Node's --check and Bash's -n all establish one thing, that the text is a well-formed program. Nothing runs, and code hidden inside a string is invisible to them. We saw this while testing ATLAS, when it edited a small web app that kept its page script inside a Python string. The Python compiled, the server started, the home page returned a success code, and the page was dead in the browser because of a stray closing parenthesis in the script.

A type checker rules out whole classes of error on every possible run, which is far beyond what a test can do. Robin Milner's 1978 theorem says "well-typed programs cannot 'go wrong'", but "go wrong" means something narrow there: an integer is never added to a truth value. A well-typed program can still compute the wrong answer. Rice's theorem (1953) explains why no static check can do more in general. Every interesting question about what a program does is undecidable, so a checker must either miss some bugs or reject some correct programs.

A build shows that the toolchain accepts the code: it compiles, links and resolves its dependencies. How much that proves depends entirely on what the build command does. For a Python project, the "build" ATLAS detects is python -m py_compile *.py, which only parses the top-level files.

A run in a sandbox shows that the code started, stayed within its limits and did not crash, on the path it took. If that path never reached the interesting code, the run says little.

A test checks the output on the inputs someone chose. The AlphaCode team estimated that, on two well-known code benchmarks, between 30% and 60% of solutions that passed all the provided tests were in fact incorrect. They also named a second failure, "slow positives", which are correct but too slow for the real limits. The EvalPlus team (2023) gave a popular benchmark eighty times as many tests, and scores fell by up to 19.3 to 28.9%. And when a model writes the tests, each expected output is a prediction, which can share the very misunderstanding the test was meant to catch.

Property-based testing, introduced by QuickCheck in 2000, states a rule and checks it on many random inputs. It finds counterexamples cheaply, and it is exactly as strong as the property. Reversing a list twice gives back the original list. So does doing nothing at all, and the property alone cannot tell them apart.

testing the tests

Coverage reports which code the tests executed. It is a good way to find untested parts and a poor measure of quality. Inozemtseva and Holmes (2014) generated 31,000 test suites for five Java systems and found only "a low to moderate correlation between coverage and effectiveness" once suite size was controlled for. A test can execute a line without checking anything about what the line did.

Mutation testing makes a small change to the program, such as turning < into <=, and sees whether any test fails. If one does, the mutant is "killed". If none does, it "survives", and its survival points to something the tests never check. The idea goes back to DeMillo, Lipton and Sayward in 1978, and Just and colleagues (2014) found that a suite's skill at killing mutants correlates with its skill at catching real faults. Some mutants can never be killed, because they behave exactly like the original, and deciding which ones those are is itself undecidable.

Luo and colleagues (2014) studied tests that are simply unreliable, with an outcome that changes from run to run on the same code. They reported Google's experience that about one test failure in twenty-two was caused by such flaky tests, and they warned that a test which fails often trains people to ignore it.

a worked example

Take this function and its test.

def median(xs):
    xs = sorted(xs)
    return xs[len(xs) // 2]

def test_median():
    assert median([3, 1, 2]) == 2

It parses, every name in it is defined, it would pass a type check, it runs without error, and the test passes. It is still wrong. For [1, 2, 3, 4] it returns 3, and the median is 2.5.

Mutation testing would have noticed. Change len(xs) // 2 to (len(xs) - 1) // 2 and the test still passes, so the mutant survives, which shows that no test uses a list of even length. Now imagine the test had been written by reading the implementation. It would assert that the median of [1, 2, 3, 4] is 3, and the bug would be locked in by its own check.

how ATLAS layers its checks

ATLAS uses several of these rungs, arranged in layers. In release 3.1.6, a write passes through them in this order.

output shape the model's reply is held to a grammar routing which writes get the full pipeline write gates syntax per language, calls to undefined names, scripts inside pages and strings pipeline checks syntax, runs and model-written tests, the build command, vetoes the gate before "done" a failing test or build sends the run back
The layers of checks in release 3.1.6.

Every reply from the model is first held to one of three structured forms. This checks form, and it makes malformed replies rare. Routing comes next: short files, configuration and documentation are written directly, and substantive code goes to the pipeline.

Before a write lands, up to three gates check it, depending on the file type and the path. A syntax gate parses the file in its own language. For Python, a structural gate rejects a call to a name that nothing defines or imports. That gate exists because an edit once called a function that had never been imported, passed the pipeline's checks, and broke every request. An embedded-script gate parses the JavaScript inside web pages, including pages kept inside Python strings, as in the web app with the stray parenthesis. The gates block an edit only if it breaks a file that passed before, and a new file must parse. Python edits are stricter: one that does not compile is always refused. The checks let the write through when the service that runs them is down or does not answer, a choice that favours availability over strictness.

Inside the pipeline, each candidate is checked according to its kind. For a Python file with no sign of an interactive program, the sandbox imports the module and runs up to five tests the model wrote from its own version of the file, and a candidate that fails more than half of them is rejected. Interactive programs, such as games and web servers, get a compile check and a lint. Most files in other languages get a syntax check and, when ATLAS detects one, the project's build command, though 3.1.6 treats C, C++, Swift, JSX, Vue and Svelte files as Python. Our note on best-of-K follows the rest of the pipeline.

The last layer is a gate before "done": when a check the agent runs fails, and nothing passed earlier in the run, the agent cannot declare the work done until one passes. Each of its checks sends a run back up to three times, and then lets it finish. In 3.1.6 an earlier pass is never taken back, and lints and plain runs of a program or a web request count as checks.

The gate before "done". In this run it sends the turn back twice, and lets it through the third time.

what release 3.1.6 leaves unchecked

Each item below is documented.

four faults in our own checks

Each of the four faults below was found and fixed on the development line, and the fixes are planned for 3.2.0.

A pipe can hide a failure. In Bash, a pipeline reports the exit status of its last command unless pipefail is set. So pytest -q | tail -n 5 succeeds even when the tests fail, because tail succeeded. Our gate saw a command that began with pytest, saw success, and counted it as verification.

A pass was never taken back. Once any check had passed during a run, a later failure did not cancel it. A run could fail, pass, fail again, and still be treated as verified.

One check could not fail at all. Our HTML syntax check used Python's standard HTML parser, which, in its own documentation's words, "does not check that end tags match start tags." It accepted any text. The same fix stopped other checkers from reporting a file as valid after running out of time or memory.

The pipeline could not see the request. It wrote its candidates and its tests from the file name, some project files and the model's own version of the file, so its tests mostly checked that version against itself.

In all four we treated a check as evidence for more than it had asked. So we want every result to say which check produced it, and what that check could and could not establish. Evidence that the program does what was asked should count for more than any number of checks that it is well formed. A degraded check should say so in its result, and a fallback should never be reported as a pass. The request has to stay in view, because a check written from the code can only confirm the code, and checking against the request often needs a person.

sources

run it yourself

ATLAS is open source under AGPL-3.0, and it runs on your own hardware.

install atlas work with us ↗