“All tests passed.” In vibe coding, that sentence can sound like a safety certificate. But a test is still code containing a question and an expected answer. If nobody checks who supplied that answer—or whether it is correct—what does the green check really guarantee?
I used a small synthetic shopping-cart app created with Codex and copies that I deliberately broke. No production service or personal data was involved. The question was narrow: When the same AI is given only the public rules, and when it is also given the clean implementation code, can the tests it writes pass the clean app, catch three pre-hidden defects, and tell apart a regression introduced by a later change?
In this article
Six steps before trusting green
I did not collapse the results into one number. I kept six observations separate:
- Was a test file generated?
- Was it executable?
- Did it pass the clean reference app?
- Did its expected answers match the answer key prepared in advance?
- Did it fail on a broken copy for the intended reason?
- Did the visible user flow reach the same verdict?
Research makes related distinctions. EvalPlus showed that adding stronger tests can reveal errors in generated code that had passed the existing evaluation. In an industrial setting that improved existing human-written tests, Meta’s TestGen-LLM treated generation, building, reliable passing, and added coverage as separate filters. UTGen/UTDebug pointed out that choosing useful inputs and producing the right expected outputs are different jobs—which is also why I checked the answer key here. I used these as methodological guidance, not as numbers transferable to this small cart.
Same model, two briefings
On September 13, 2026, I ran gpt-5.6-luna through codex-cli 0.153.4 twice at low reasoning effort. The spec-first condition received only the public rules and function descriptions. The code-aware condition received the same material plus the clean implementation. The model and prompt were identical, and each condition generated its tests once in a fresh session.
The generator never received the private answer key, D1–D3 defect list, or the later note feature. It ran from a temporary directory while an operating-system policy denied reads and writes under the repository paths that held the hidden material. Browser, web search, and plugins were also turned off. The existing authenticated service connection remained in use, however, and startup warnings showed that some user-level CLI state metadata was consulted. Full isolation from unrelated user-level metadata was not independently proven. The first two launch methods failed before the model ran because of isolation-setup problems; those failures remain in the record. Only the third method completed both conditions.
The rules-only tests failed on the clean app
The spec-first run generated five tests. The syntax check passed and the suite executed, but only two tests passed on the clean reference app; three failed. It stopped at the clean-pass step of the six-part check.
Most of the arithmetic was correct. The problem was the stored coupon string. The public rules required ignoring surrounding spaces and comparing SAVE10 without regard to letter case, but did not require rewriting the stored input in uppercase or clearing it when the final item was removed. The generated tests wrote both behaviors into their expected answers. The reference app preserved the original input.
Under the protocol’s rules, that makes the suite suite-invalid: a void condition because the tests did not pass the reference app. I did not proceed to D1–D3; a red result on a broken copy could not count as catching a bug when the tests were already red on the clean app. Generating a test and generating a usable expected answer were visibly different outcomes.
The tests that also saw the code caught all three hidden defects
The code-aware run generated eight tests, and all eight passed the clean reference app without editing expected values or checks. An audit of its 16 assertion calls found that eight were independently derivable from the public rules; the other eight were tied to the current object shape or implementation-observed storage. Green alone did not make every expected answer independent of the implementation. I then introduced the hidden defects one at a time.
- D1 used the pre-discount subtotal for free shipping. The shipping-and-total test failed.
- D2 made coupon matching case-sensitive. The sAvE10 discount test failed.
- D3 retained the old total after removing the final item. The reset-to-zero total test failed.
Each failure matched the intended value in the private answer key. A visible preflight had separately checked the clean reference flow. Still, this does not mean the AI found three unfamiliar real-world bugs. D1–D3 were known rule defects that I chose and hid before generation. Nor can I say that showing the code caused better tests: underspecified storage behavior can also explain part of the difference. This is one observation from one model, one prompt, and one run, and three selected seeds do not estimate a general bug-detection or safety rate.
With a new feature, “too exact” tests broke on the clean change
I kept the eight code-aware tests unchanged and added an item-note feature. In the clean changed app, removing the final item also removed its note. The regression copy, LR1, reset every amount to zero but left the deleted item’s note on screen. The private answer key and visible flow distinguished them: the clean change passed, while LR1 failed.
The unchanged code-aware suite did not. It failed the same five of eight tests on both versions because the tests compared the whole cart state exactly and treated the legitimately added note field (itemNote) itself as an error. Since the clean changed app was already red, the run could not reach a valid fault-discrimination verdict: LR1 is inconclusive, not detected.
That is not merely noise. An expected answer copied too tightly from today’s data shape can catch a known defect and still become brittle under a legitimate extension. A regression test has to do more than turn the light red; it has to tell us why the light came on.
Five questions to ask next
If AI has written the tests, these five questions are a good place to start:
- Does it first pass a clean baseline case?
- Were expected values derived from requirements rather than copied from code?
- Does the intended check fail when one behavior is deliberately changed?
- Does it test user-visible outcomes instead of the entire current object shape?
- Do the automated result and the visible flow agree?
Generation took 42.26 seconds for spec-first and 49.24 seconds for code-aware. No extra purchase was made. The command-line tool logged token-usage signals but did not show a per-run price or subscription-quota deduction, so I did not record the cost as zero. The test files were preserved as plain .mjs files.
The conclusion is neither “don’t trust AI tests” nor “show the model the code and the problem is solved.” Don’t treat a pass as the end of trust: check the expected answers, whether faults are told apart, whether tests survive legitimate change, and the on-screen flow—each separately. In this small experiment, the spec-first suite became invalid at the clean-app step. The code-aware suite caught three selected hidden defects, but its shape-bound checks broke on a legitimate feature addition and left the later regression inconclusive. These are two traps from one run, not a general safety or bug-detection rate.
AI was used to assist with research and drafting. The author independently verified and edited the final article.
답글 남기기