A broken export passed all three tests. Here is why.

The faulty version returned two records:

a-complete
b-complete

The request was for company A. The second record belonged to company B. All three original checks passed.

This was a deliberately constructed, executable demonstration, not a customer incident or a test of an AI model. Its purpose was to make one blind spot visible without asking the reader to trust a story about it.

The tests were working. They were answering the wrong question.

The experiment contains three synthetic records: a completed job for company A, an open job for company A, and a completed job for company B. The correct export returns only a-complete.

The initial checks ask whether the result contains rows, whether each row has the expected columns, and whether every returned job is completed. These are legitimate checks. None asks which customer owns the jobs.

Here is the condition that creates the two versions:

.filter((job) => removeCustomerBoundary || job.company === company)

With removeCustomerBoundary set to false, the customer filter applies. With it set to true, every completed job passes through. The second setting deliberately removes the boundary; it is not a proposed implementation.

Running both versions produced this result:

Version	Records returned	Original checks passed	Exact-record check
Customer filter present	a-complete	3 of 3	Passed
Customer filter removed	a-complete, b-complete	3 of 3	Failed

The original checks were not malfunctioning. Both outputs satisfied everything they had been asked to verify. Their passing status was real, but it did not answer the question that mattered for this request.

A second reviewer needs something different to compare

Imagine asking a second reviewer to confirm that the export contains rows, has the expected columns and includes only completed jobs. The broken version still satisfies that description. Repeating an incomplete acceptance condition does not make it complete.

This experiment did not run two agents, so it cannot establish how often agents would make that mistake. It demonstrates the underlying limitation: agreement on these three properties cannot establish customer separation.

The additional check compares the returned identifiers with an independently specified expectation:

assert.deepEqual(rows.map((row) => row.id).sort(), ['a-complete']);

That expectation comes from the requirement and the fixture, not from asking the export function to generate its own expected answer. The deliberately broken version fails it.

This is the difference I would look for in a verification proposal. What information, requirement or observation does the verifier have that could contradict the implementation? “Another agent reviewed it” does not answer that question by itself.

The experiment also puts a limit on its own result

The stronger check catches this defect in this fixture. That is the finding.

It does not prove that every export is secure. The demonstration does not exercise HTTP requests, authentication, CSV serialization, download links or a production database. It says nothing about model reliability, and one detected defect is not a general detection rate.

There is another boundary worth keeping visible. A model saying “the tests passed” is a report about execution. The evidence is the actual execution record, the checks that ran, their results and the version they examined. The downloadable result here records the script’s hash and runtime alongside its output.

Those distinctions are not reasons to avoid automated development. They make it possible to assign automation a useful role: produce the change, run the checks, expose the evidence and let a conflicting result stop the release.

Run it and inspect the difference

The standalone script has no dependencies or network calls. Save it as experiment.mjs and run:

node experiment.mjs
node experiment.mjs --json

The recorded output includes the synthetic inputs, both results, the source hash and the limits of the demonstration. The runner exits successfully only when the correct version passes and the additional check rejects the deliberately broken version.

For an actual project, the next question is not “How many reviewers do we have?” It is: what plausible wrong result would make our current verification fail?

That is a more useful starting point for independent verification than adding another approving voice. The next article in the series examines which access a coding agent actually needs.