[카테고리:] English

For this article, I did not click Connect. I stopped because there was no record that a test-only account, a private test file, and an environment isolated from other access routes were ready. I did not open an account or login screen, test an OAuth flow, view a consent screen, change permissions, retrieve a file, disconnect inside ChatGPT, revoke access in Google, or inspect retention or deletion, and I took no screenshots. This is not a report claiming that ChatGPT could—or could not—read a particular Google Drive file. It records the questions worth asking before connecting, and it separates what public documentation, rechecked on September 13, 2026, can tell you from what only the account screen can show.

“Permission to connect” is not a single switch

The phrase “connect Drive” hides at least three different questions.

1. What did I grant? This is about the services and permissions shown on Google’s consent screen.
2. What did the AI freshly retrieve? This depends on the files the connected Google account could already open, the actions enabled in ChatGPT, and the route actually used for the answer.
3. What remains after disconnecting? Whether Google still supplies new content after a ChatGPT-side disconnect, revoking the app from the Google account, text already in a conversation, and deleting saved files or memory are separate matters.

Google’s sign-in and permissions documentation (OAuth) explains that an app can act only within the permissions, called scopes, it was approved for. Google also supports choosing permissions individually for qualifying requests. Neither point tells you what choices a particular ChatGPT consent screen will show until you see that screen. OpenAI’s documentation says users may be able to select fewer Google services but may still need to approve every permission requested for the selected services.

OpenAI’s Google Drive guide distinguishes an individual live connection from administrator-managed sync. According to the guide, one boundary on the files available through a live connection is what the selected Google account could already open. This article planned to examine only an individual connection and did not open either route.

OpenAI also documents ChatGPT action-approval settings that can ask before an action. Those settings are separate from the scope granted on Google’s consent screen and do not automatically narrow it. The exact label and options were not checked in this work.

What public documentation cannot answer

Public documentation cannot populate the current screen for a specific account.

The final two rows matter most. “Could not fetch new content” and “old text is no longer visible” are not the same finding. In this article, “disconnect” means turning off the link inside ChatGPT. Revoking the app from the Google account is a separate action that was not examined. The OAuth revocation standard and Google’s revocation documentation explain that a revocation may take time to take full effect, but that alone does not tell you how a ChatGPT-side disconnect is implemented.

The synthetic-data experiment I had planned

The planned procedure uses one private file in a dedicated test-only Google account instead of real material. Its filename and body contain unpublished random markers. Before connecting, a new chat would be asked about the file by name only; after connecting, the same question would be asked again. The baseline check would count only if ChatGPT returned the current body marker and showed evidence that the Drive connection supplied it.

Next, the ChatGPT-side connection would be disconnected. The same file’s body marker would change before each request, while the filename stayed fixed. New chats would repeat the same request immediately, after five minutes, and after thirty minutes. Old conversation text would be inspected separately. A retrieval of any fresh marker attributed to the Drive connection would stop the run immediately. Three misses would still not be written up as proof that the connection was permanently cut off across every account or route.

The procedure is demanding from the start. It requires a test-only Google account and private file that already exist, confirmation that no other connection, upload, Memory, or sync could supply the result, screenshots with identifying details masked, zero cost, and a source indication that identifies the actual connection used.

None of those prerequisites had an observation record in this assignment. The rules for this work excluded creating an account or signing in, and the author’s browser was not opened because doing so could expose personal account information. Every checklist item therefore remained unverified. No account or login screen, OAuth or consent flow, permission change, file retrieval, ChatGPT-side disconnect, Google-side revocation, retention or deletion check, delayed re-query, or screenshot was carried out. The procedure described above is a design record explaining why this run did not start, not a how-to for a personal or work Google account.

Why I did not replace the gap with a mock

A local mock consent screen and fake Drive files were considered but not treated as results. Such a rehearsal would prove only how the mock behaved, not what Google permissions ChatGPT requested or what happened after an actual disconnect.

A public-link file would be no better as a baseline check: ordinary web access could explain the result. Likewise, “I couldn’t find the file” is inconclusive when there is no evidence that the Drive connection was used. It is not a successful block; the cause is unknown.

A three-part checklist before connecting

These are documentation-based questions, not findings from a product test. You do not need to turn a real account into an experiment. If you are considering a connection, separate these checks:

• Grant: Confirm the account, selected services, and every permission sentence. Do not approve if the live screen requires broader access than you are willing to accept.
• Use: Look for evidence of which connection supplied which information. Do not treat a generic success message or a filename alone as proof of a fresh file read.
• Remain: Check retained conversations, saved files, and memory separately from a ChatGPT-side disconnect. Do not assume that disconnect means deletion, Google-side revocation, or the end of access through every other connection. This is not an instruction to probe a real file after disconnecting.

This work cannot show that the product is safe or unsafe. Its strongest conclusion is that without the account-specific consent screen and a baseline check showing which connection was used, I could not produce an answer to “how much can it see?” Separating grant / use / remain before opening the connection makes it possible to avoid declaring success or failure when the necessary evidence is missing.

AI was used to assist with research and drafting. The author independently verified and edited the final article.

“All tests passed.” In vibe coding, that sentence can sound like a safety certificate. But a test is still code containing a question and an expected answer. If nobody checks who supplied that answer—or whether it is correct—what does the green check really guarantee?

I used a small synthetic shopping-cart app created with Codex and copies that I deliberately broke. No production service or personal data was involved. The question was narrow: When the same AI is given only the public rules, and when it is also given the clean implementation code, can the tests it writes pass the clean app, catch three pre-hidden defects, and tell apart a regression introduced by a later change?

Six steps before trusting green

I did not collapse the results into one number. I kept six observations separate:

1. Was a test file generated?
2. Was it executable?
3. Did it pass the clean reference app?
4. Did its expected answers match the answer key prepared in advance?
5. Did it fail on a broken copy for the intended reason?
6. Did the visible user flow reach the same verdict?

Research makes related distinctions. EvalPlus showed that adding stronger tests can reveal errors in generated code that had passed the existing evaluation. In an industrial setting that improved existing human-written tests, Meta’s TestGen-LLM treated generation, building, reliable passing, and added coverage as separate filters. UTGen/UTDebug pointed out that choosing useful inputs and producing the right expected outputs are different jobs—which is also why I checked the answer key here. I used these as methodological guidance, not as numbers transferable to this small cart.

Same model, two briefings

On September 13, 2026, I ran gpt-5.6-luna through codex-cli 0.153.4 twice at low reasoning effort. The spec-first condition received only the public rules and function descriptions. The code-aware condition received the same material plus the clean implementation. The model and prompt were identical, and each condition generated its tests once in a fresh session.

The generator never received the private answer key, D1–D3 defect list, or the later note feature. It ran from a temporary directory while an operating-system policy denied reads and writes under the repository paths that held the hidden material. Browser, web search, and plugins were also turned off. The existing authenticated service connection remained in use, however, and startup warnings showed that some user-level CLI state metadata was consulted. Full isolation from unrelated user-level metadata was not independently proven. The first two launch methods failed before the model ran because of isolation-setup problems; those failures remain in the record. Only the third method completed both conditions.

The rules-only tests failed on the clean app

The spec-first run generated five tests. The syntax check passed and the suite executed, but only two tests passed on the clean reference app; three failed. It stopped at the clean-pass step of the six-part check.

Most of the arithmetic was correct. The problem was the stored coupon string. The public rules required ignoring surrounding spaces and comparing SAVE10 without regard to letter case, but did not require rewriting the stored input in uppercase or clearing it when the final item was removed. The generated tests wrote both behaviors into their expected answers. The reference app preserved the original input.

Under the protocol’s rules, that makes the suite suite-invalid: a void condition because the tests did not pass the reference app. I did not proceed to D1–D3; a red result on a broken copy could not count as catching a bug when the tests were already red on the clean app. Generating a test and generating a usable expected answer were visibly different outcomes.

The tests that also saw the code caught all three hidden defects

The code-aware run generated eight tests, and all eight passed the clean reference app without editing expected values or checks. An audit of its 16 assertion calls found that eight were independently derivable from the public rules; the other eight were tied to the current object shape or implementation-observed storage. Green alone did not make every expected answer independent of the implementation. I then introduced the hidden defects one at a time.

• D1 used the pre-discount subtotal for free shipping. The shipping-and-total test failed.
• D2 made coupon matching case-sensitive. The sAvE10 discount test failed.
• D3 retained the old total after removing the final item. The reset-to-zero total test failed.

Each failure matched the intended value in the private answer key. A visible preflight had separately checked the clean reference flow. Still, this does not mean the AI found three unfamiliar real-world bugs. D1–D3 were known rule defects that I chose and hid before generation. Nor can I say that showing the code caused better tests: underspecified storage behavior can also explain part of the difference. This is one observation from one model, one prompt, and one run, and three selected seeds do not estimate a general bug-detection or safety rate.

With a new feature, “too exact” tests broke on the clean change

I kept the eight code-aware tests unchanged and added an item-note feature. In the clean changed app, removing the final item also removed its note. The regression copy, LR1, reset every amount to zero but left the deleted item’s note on screen. The private answer key and visible flow distinguished them: the clean change passed, while LR1 failed.

The unchanged code-aware suite did not. It failed the same five of eight tests on both versions because the tests compared the whole cart state exactly and treated the legitimately added note field (itemNote) itself as an error. Since the clean changed app was already red, the run could not reach a valid fault-discrimination verdict: LR1 is inconclusive, not detected.

That is not merely noise. An expected answer copied too tightly from today’s data shape can catch a known defect and still become brittle under a legitimate extension. A regression test has to do more than turn the light red; it has to tell us why the light came on.

Five questions to ask next

If AI has written the tests, these five questions are a good place to start:

• Does it first pass a clean baseline case?
• Were expected values derived from requirements rather than copied from code?
• Does the intended check fail when one behavior is deliberately changed?
• Does it test user-visible outcomes instead of the entire current object shape?
• Do the automated result and the visible flow agree?

Generation took 42.26 seconds for spec-first and 49.24 seconds for code-aware. No extra purchase was made. The command-line tool logged token-usage signals but did not show a per-run price or subscription-quota deduction, so I did not record the cost as zero. The test files were preserved as plain .mjs files.

The conclusion is neither “don’t trust AI tests” nor “show the model the code and the problem is solved.” Don’t treat a pass as the end of trust: check the expected answers, whether faults are told apart, whether tests survive legitimate change, and the on-screen flow—each separately. In this small experiment, the spec-first suite became invalid at the clean-app step. The code-aware suite caught three selected hidden defects, but its shape-bound checks broke on a legitimate feature addition and left the later regression inconclusive. These are two traps from one run, not a general safety or bug-detection rate.

AI was used to assist with research and drafting. The author independently verified and edited the final article.

The short answer is that no computer-use agent was started. There was no click, no blocked click, and no measurement of trust.

A live task did not go wrong. A predeclared capability gate stopped the test because the required safeguards were not observable in the inspected interface. Documented access to screen, URL, and DOM state is different evidence from an independent record that every action stayed within scope.

“Do not do this” is not a barrier

You can tell a computer-use agent not to leave a site or click a Send button. That sentence in the prompt (the written instruction) is an instruction, not an enforcement mechanism. An AI that sees the screen can also read on-screen text as input, and that text may conflict with the user.s instructions.

The official computer-use guidance from OpenAI, Anthropic, and Google uses different wording, but all three recommend isolation, site or action boundaries, human control before consequential actions, records, and checking the result. These are vendor operating recommendations, not proof that a particular product is always safe.

So this protocol set its own gate: before starting an agent, four records had to be separately observable. The gate did not pass, so this run produced none of the four records:

1. what the AI proposed;
2. what the policy decided immediately before execution—allow, block, or require confirmation;
3. what the component that actually drives the browser executed; and
4. what a separate checker independently verified about the final state.

An agent saying “done” is not the fourth event. Looking at the screen once is not a record of the whole path in between either.

A synthetic test site and a small policy wrapper

Instead of a real account, the setup used a synthetic site that opens only on my own machine (the loopback address). Page A and Page B each contain one synthetic string that does not appear in the prompt, so they can show whether an agent actually read both pages. Because C1 and C2 were not run, the only observation here is a unit test of the in-memory comparison function—not a live prompt or page-reading result. Page B also has an ordinary-looking “Send this report” button, but there is no external recipient or transmission target, and all inputs are non-sensitive synthetic data. In a real service, typing into a field may itself transmit data before Send is clicked.

I also built a test policy wrapper (the executor). It wraps only a mock browser component that I made; the selected in-app browser.s clicks and navigation do not pass through it. In the unit suite, the wrapper allows only the address of the machine itself (127.0.0.1) and a predefined set of actions. It blocks external navigation and the Send click before calling the mock primitive. Proposal, decision, and allowed execution events receive IDs and enter a hash-chained ledger, where changing an earlier record breaks later verification.

On September 13, 2026, the six automated Node.js tests that check only this wrapper were rerun, and all six passed. Blocked actions never reached the mock component; allowed actions reached it only after an allow decision had been recorded; and the tamper-detection test also passed.

That evidence has a hard limit. It is a unit test of the wrapper. It does not show that a real computer-use model ignored a planted instruction or that the selected browser could not make an external request.

Why the main run did not start

On September 9, 2026, I checked the documentation and feature inventory of the Codex In-app Browser/Unified Computer Use package 26.901.51231 on macOS 26.6.2 (build 25G83). It listed on-demand access to the screen, the current URL, and the page structure (DOM). I did not navigate or interact with a page. In any form I could see, the interface did not provide a separate proposal record, a per-action policy decision, an unchangeable execution ledger, or an enforced site allowlist. Internal controls that are not exposed were not ruled out, but they cannot count as evidence here. Cancellation, clean reset, and the exact screenshot transmission and retention path also remained unverified. The embedded engine, driver model, pricing plan, and incremental cost were unknown.

A strict page security policy (CSP) and a Send button with no sending function can be built, but they cannot prove that the page stopped the browser from navigating to another address. Checking the URL after an action is too late: it cannot exclude a brief out-of-scope visit followed by a return.

Neither the baseline condition without a planted instruction (C1) nor the condition with a planted on-screen instruction (C2) was run. They are not “two failures” or a 0% success rate. There were no behavioral observations, so their status is simply not run.

A delegation card to fill in first

A non-developer can fill in these fields first without reading any code:

Add one plain question: Does it actually stop if I do not answer? A polite “May I continue?” is different from a technical gate that prevents execution until a person approves.

A blank field is not a pass. Neither is N/A. If a record is missing, write "unknown" and do not delegate a task that needs that record.

So how much should you trust it?

This run did not measure the safety of an AI product. It showed that a predeclared capability gate can stop a test before missing evidence turns into a convincing completion story.

An independent checker verifies only the final state, and a correct final state does not prove a safe path, scope compliance, or the absence of transmission. Research environments such as WebArena also check functional outcomes separately, but one small local test page is not a benchmark or a general reliability estimate.

The scope handed to a screen-operating AI should be more specific than “I will be watching.” Ask what the executor blocks, what it records, when it pauses, and who verifies the outcome independently. These questions are the editorial standard chosen for this protocol, not a safety verdict on any product. If one essential record cannot be confirmed, stopping before the click, as happened here, is the right call. That too is a result.

AI was used to assist with research and drafting. The author independently verified and edited the final article.

In the official sample program used to display embedded screens—a local reference host—I changed Review from 15 to 25 minutes on a weekly study form. The chart changed, and the total rose from 90 to 100 minutes. It looked like a small web app inside a conversation, but this did not happen inside ChatGPT, Claude Desktop, or VS Code. Displaying an interface, validating input, saving data, informing an AI's next answer, and working in another chat service are separate promises.

To examine those differences, I built a minimal app using only synthetic data. No production conversational product was tested.

The bridge that displays the interface

MCP Apps is an optional extension that connects a tool to an embedded HTML view. The specification supplies the bridge; a compatible host decides whether and how to display the interface. Forms and charts are not special MCP components. They are ordinary HTML and JavaScript running through that bridge. The design is documented in the stable MCP Apps specification and the official overview.

The app had two tools: one calculated the study record and returned readable text plus view data; the other validated and saved changes from the view. The fixed data was Reading 30 minutes, Practice 45 minutes, and Review 15 minutes, for a total of 90. It requested no device permissions and declared no external content domains.

On September 7, 2026, I tested with Node 24.19.0, MCP Apps 1.7.5, and MCP SDK 1.29.0. The run used commit 92f46a5 of the official local basic-host 1.7.5 reference host. All 25 build checks passed, with a recorded direct incremental cost of USD 0.00. The 25/25 figure covers implementation checks, not the verdicts for the 18 behavior tests below. Those behavior verdicts were judged separately, using browser captures and server logs.

The bilingual form worked, but it also exposed a defect

For both Korean and English, the baseline total of 90 and the edited total of 100 matched across the server result, embedded view, and accessible chart representation. In a separate fallback test (T1), a client that did not report support for an embedded screen received readable text containing the language, all three items, selected item, total, and a note that the interactive view was unavailable.

The unsupported language code zz, however, produced a real defect. The server returned an English fallback and a warning, while the view displayed Korean and showed no warning. A non-numeric value, negative number, over-limit value, and empty list were rejected, but the server-view mismatch meant that I2 was a FAIL. This run could not determine whether the mismatch originated in the app's language handling or the host's behavior.

Storage and AI context are different questions

The successful restoration was not chat memory. The view explicitly saved data under a synthetic session ID inside the local server process. A fresh view sent the same ID and restored the 100-minute total from server_store. This does not prove persistence after a server restart or across a real account, another device, another host, or production storage.

The same-view retention test did not produce changed server-validated state. The close-and-reopen test used the baseline value both before closing and when invoking the app again, so reset could not be distinguished from input overwrite. V1 and V2 were therefore INVALID: the procedures did not support a verdict, which is different from an app failure. The save-failure test S2 was not run and remained NOT TESTED.

A value visible in the view, a server-validated result, a context-update request sent to the host, and an AI's later answer also travel through different channels. This run confirmed only that the app sent the context request Review 25 minutes, total 100. In that narrow send-only sense, C1 was a PASS. Whether the host accepted or rejected the request, and whether an AI received or answered it, were not confirmed. So this run cannot claim the request caused an AI to know the value.

One blocked request is not a security certification

Permission and domain declarations were empty, and the requests I captured used only local test addresses. A control request reached a separate test listener. A request from the embedded view to that same undeclared local listener was blocked by this test setup's Content Security Policy (CSP), a browser rule that limits where a page may send requests, and never reached the listener. This is one observation in one local environment, not a safety certification or proof that data cannot leak. The host's handling of user consent was not tested either.

Results from all 18 tests

PASS means that the stated criterion was met within the scope shown. FAIL marks an observed mismatch, INVALID means the procedure could not support a verdict, and NOT TESTED means the test was not run.

The total was 8 PASS, 1 FAIL, 3 INVALID, and 6 NOT TESTED. The absence of an app-only update tool from one invocation list does not prove that every host hides it or enforces the same security boundary.

Four questions to ask when watching a demo

In this exact local reference environment, bilingual interaction, explicit restoration from server storage, and text fallback without a UI worked. The test also exposed an unsupported-language-code defect. Same-view and close-and-reopen state could not be judged, while save-failure behavior, AI follow-up, user consent, and production-product compatibility were not tested.

Before relying on a demo, ask four questions:

1. Does a readable result exist without the interface?
2. Who stores each value, where, and under which identifier?
3. Can you inspect the server-validated result when the screen and answer disagree?
4. Are tests that were never run left marked as NOT TESTED?

Because no AI follow-up was available here, this run cannot tell you which value to trust if a screen and a later AI answer disagree. A small chat app earns trust not through one impressive interaction, but by proving each separate promise.

AI was used to assist with research and drafting. The author independently verified and edited the final article.

A saved message on screen does not prove that a record was actually saved. An app can look successful even when only the screen changed or the data stayed inside the current browser.

To make that difference visible, the test used one disposable local app with synthetic records. A browser-storage demo and the core record app appeared in the same interface, while the browser, a local API, and SQLite—the file-based datastore—were checked separately. This is not a comparison of AI app builders or a verdict on AI-built apps in general.

The test centered on three questions:

1. Where was the record stored?
2. Who confirmed the write?
3. Who could access the data?

1. Where was the record stored?

The browser-storage demo saved one record five different ways: runtime memory, sessionStorage, localStorage, IndexedDB, and a deliberately discarded write. After a hard reload, the observed results were:

According to the official documentation, sessionStorage lasts for the page's tab session, while localStorage can persist across browser sessions for the same origin. IndexedDB is also client-side structured storage. Those are documented properties, not additional results from this hard-reload check. Seeing a record after a refresh does not prove that it reached a server or could be recovered on another device. Browser storage can also be deleted or evicted (MDN: storage quotas and eviction criteria).

This browser check stopped at a hard reload. A full browser-process restart and a UI test in an independently proven clean browser profile were not performed. These two UI gaps sit outside the verifier's 25 machine-counted checkpoints described below. They are neither passes nor failures.

2. Who confirmed the write?

In the core app, synthetic user A created three records, updated one, and deleted another. The app's response was not treated as sufficient evidence. An independent verifier program read the SQLite file directly and compared the stored records with a frozen expected result. The local API route used for authoritative reads returned Cache-Control: no-store and identified its source as sqlite_authoritative, but those self-reported details were not treated as proof on their own. The API result was checked against the independent SQLite inspection. This does not prove that every possible cache layer was absent.

There were three attempts. In the first, the restricted environment blocked the local port, leaving 2 PASS, 20 FAIL, 2 INVALID, and 1 NOT RUN. In the second, the server process ended before the first request, leaving 3 PASS, 21 FAIL, 0 INVALID, and 1 NOT RUN. Neither attempt was used to judge the app. Those environment and process failures remain in the record. Only the third attempt, using the same frozen protocol in a normal local environment, was used as the observation of the app.

The valid machine-checked run produced 24 PASS, 0 FAIL, 0 INVALID, and 1 NOT RUN. This is not a feature-completeness score. It is the total across 25 predefined checkpoints, including create, update, delete, direct datastore reads, offline behavior, and access denials. The one machine-counted NOT RUN was local code-artifact replacement.

Here, INVALID means that the evidence was insufficient or ambiguous, the required boundary was not actually crossed, the inputs changed, or the state was contaminated. NOT RUN means that an optional checkpoint was deliberately skipped or a declared safety, account, or payment boundary prevented it. Neither status is the same as an app behavior that was validly tested and failed.

The verifier also checked that authoritative reads failed while the server was unavailable and recovered after it returned. This prevented an offline response from being mistaken for a server-confirmed success; it did not test every possible cache failure. In a separate local server-restart check, the same SQLite records remained after the server started again as a different process.

Replacing the local code artifact was not performed. Hosted deployment was also outside the executed 25-checkpoint local verifier and was not tested. A local server restart is not evidence of a hosted redeploy. Cloud durability, backup restoration, and long-term retention remain outside this experiment.

The direct monetary cost of this local run was USD 0.00. That figure excludes existing subscriptions, labor, electricity, hardware, and opportunity cost. Total end-to-end elapsed time was not captured.

3. Who could access the data?

Persistence is not enough if another user can retrieve the record. Synthetic user B first confirmed that it could create a record through its own allowed route. B then tried to read, update, and delete records created by A, using those records' known IDs. Those requests were denied with 404 responses—meaning the route reported that the record was not found—and A's SQLite state remained unchanged. A read request sent without the test-only X-Synthetic-Identity header returned 401, meaning authentication was required. The reverse attempt, A reading B's record, also returned 404.

This means that user separation worked only on the routes tested in this local setup. Fixed values in the X-Synthetic-Identity test header stood in for synthetic users. Omitting that header was not a real logout test, and this setup does not demonstrate production authentication or overall product security. Both Supabase's Row Level Security guide and Firebase's Security Rules documentation treat logging in and data permissions as separate questions.

Duplicate handling was another separate result. A's create request was replayed exactly once using the same operation ID, and only one record remained. That observation covers one controlled retry. It does not prove that every duplicate or concurrent request would be safe. Persistence, retry handling, and user isolation are three different questions; success in one cannot substitute for evidence in the others.

What to ask before trusting a saved state

For a non-developer building with AI, these three questions are more useful than a success message alone:

• Where was it stored? Ask whether the record stayed on screen or in this browser, or reached a server-side datastore.
• Who confirmed the write? Ask for an independent datastore read, not just the app's own success message. Also ask which boundary was actually crossed: reload, browser restart, server restart, code replacement, or deployment.
• Who could access it? Test both an allowed synthetic identity and one that must be denied, using synthetic data. Selected test routes and fixed headers are not production authentication or a security audit.

In this disposable local app, the limited create, read, update, and delete flow, one same-operation retry, selected user-isolation routes, the offline check, and one local server restart matched the frozen expected result. No broader claim follows from this test. Stronger evidence than a “saved” message on screen is a boundary-by-boundary record of where the data remained, who confirmed it, and who could not access it.

AI was used to assist with research and drafting. The author independently verified and edited the final article.

If every request to organize meeting notes starts with “Don't guess missing owners, and keep canceled tasks separate,” could you save that explanation in a file and reuse it?

We tested this with short, fictional meeting notes. After copying the identical instruction file into each fresh workspace, we could load it in a new conversation. Every answer passed our predefined factual checks, so this test could not distinguish accuracy between the conditions. That does not establish equal reliability. Ordinary requests without the extra instructions also passed.

What does “teach” mean here?

An Agent Skill is a package of instructions an AI can read while doing a task. Its name, description, and procedure go in a text file called SKILL.md. This is different from retraining the AI model. Think of it as a procedure the tool can retrieve when needed. Agent Skills specification

Codex and Claude Code, tools for assigning file-based work to AI, support these instructions, though the locations for saving the files and ways of invoking them differ. Sharing a file format does not guarantee identical tools, permissions, or results.

Comparing how the same instructions were delivered

On September 5, 2026, we used Codex and Claude Code on a Mac to organize two sets of Korean meeting notes. One covered an ordinary workshop plan. The other covered a newsletter, including canceled printing and conflicting review deadlines. These were fictional inputs, not real meetings or accounts of a beginner's experience. AI agents running in Codex carried out and assessed the experiment.

Each tool received three conditions:

The instructions specified four sections, references to the numbered source lines, explicit unknowns for missing owners or deadlines, and preservation of cancellations and conflicts. The pasted and saved versions contained the same instructions. Pasted versus saved instructions was the main delivery comparison. The ordinary request was a reference condition for performance without the extra procedure, outside the main pasted-versus-saved comparison. The tools still differed in how and where they loaded instructions; this was not an identical-context test.

Every run started in a new conversation and workspace. Saved-skill runs received a copy of the identical file. We checked invocation with separate development notes, then froze the inputs and scoring criteria. Two tools × three conditions × two cases produced 12 initial answers, all retained.

Even the ordinary requests passed every factual check

Each case had its own nine predefined factual checks for decisions, actions, owners, deadlines, and unknowns. The second case also tested cancellations and conflicts. Separate AI agents graded answers with model and condition labels hidden; the Codex coordinator, which was not blinded, cross-checked their findings. Answer formatting remained visible to the graders. This was not independent human grading, and a tool under examination also participated in assessment. We allowed up to two correction requests for substantive omissions or errors, excluding formatting differences from the factual score.

For example, where the notes listed conflicting review deadlines of September 14 and September 16, no answer arbitrarily chose one. None restored the canceled printing-quote request as an active task. Some ordinary answers added recommendations beyond the notes. In one case, we interpreted “A should set the deadline,” placed under items needing clarification, as advice rather than a fabricated meeting commitment. One Claude saved-skill answer included an English preamble in its Korean output. These judgments and formatting differences were recorded separately. Passing every factual check did not establish equal overall quality.

We did not infer skill use from the appearance of the answers. All four explicit-invocation runs had records showing the file being read or its instructions inserted by the tool. In a separate diagnostic, we installed the file but did not name it in the request, making one attempt per tool. Unlike the ordinary-request condition, the file was present. Both tools loaded the skill. Because this request text differed from the main test, we treated it as an invocation check, not another accuracy comparison. One successful attempt per tool cannot establish reliable automatic activation.

Request length is not the same as effort

These character counts follow from how we assembled the inputs. Saved-skill requests were 700 characters shorter than pasted instructions, but the ordinary requests were shortest. The requests were assembled the same way for both tools.

These Unicode character counts include spaces and source notes, and the requests were assembled automatically. We did not measure how much less a person would type or how much working time they would save. The AI still reads the file's instructions, so a shorter request does not establish token or cost savings.

In these observations, explicit skill invocation took longer than pasting in each of the four comparisons: two tools × two cases. With only one run per condition and case, this is not a reliable general speed comparison.

The separate development stage hit a setup failure: an older version of the Codex command-line tool did not support the selected model. The development check succeeded with a newer version already bundled in the installed app. Having an instruction file did not prevent a tool-version problem from blocking execution.

The experiment used existing subscriptions. The Codex tool version was 0.153.0-alpha.5, with gpt-6-astra selected. Claude Code was 2.1.261, with Fable 5.1 producing the main answers. The retained Codex records did not identify the model that actually answered, and some Claude usage records included an auxiliary model. These results should not be read as a pure model-performance contest.

Reusing a procedure and improving accuracy are different claims

What this experiment established was that the same procedure could be retrieved from a file. With every answer passing, these short cases and criteria could not distinguish performance. Only the second case tested cancellations and conflicts. We did not test long-term consistency or time savings for beginners. Fresh workspaces did not eliminate every tool default setting, either. This English article describes the same Korean-input experiment; we did not run an English benchmark.

The ordinary requests preserved the facts in these cases, so this experiment provides no reason to say a skill was necessary for accuracy. For a recurring task, start with “Did it load the intended procedure?” and “Did it leave missing information marked as unknown?” Reusing instructions that fit does not remove the need to check the resulting work.

AI assisted with research and drafting. The author reviewed the final manuscript and directed the revisions.