AI answers with citations: will they stop where the documents stop?

A fictional equipment-sharing club's January policy allows seven-day borrowing periods. A replacement policy takes effect in July and extends them to fourteen days. If you ask about the current borrowing period, is an exact quotation of the old seven-day rule enough? The wording is real, but it does not answer the question for the date that matters. All dates and policy values are invented. One model received six documents and twelve paired Korean/English questions, once per language and condition. No current-period answer used the superseded seven-day rule in this run, but some correct answers had inadequate quotations.

The test separates three checks: whether a source exists, whether its quoted passage supports the claim, and whether the rule applies to the question's date, item and scope of exceptions.

Two distinctions borrowed from earlier evaluations

The ALCE study evaluates answer correctness separately from citation quality. The SQuAD study on unanswerable questions examines answering when evidence exists and abstaining when it does not. SQuAD is an extractive reading task, unlike this generative-answer test. The borrowed ideas are narrower: separate correctness from citation quality, and answering from abstention. Dates, exceptions and conflicts are distinctions in this experiment’s design. The studies do not establish this model’s performance.

The experiment: the same documents, two requests

On September 27, 2026, the experiment used six synthetic documents and an answer key written before the runs. The documents covered the old and current policies, a camera exception, pickup instructions, and two conflicting projector-fee notices. Every request received all six short, deliberately legible documents in the same order. Searching for documents and order effects were not tested. The current policy explicitly replaces the old policy and retains the camera exception. The planning draft’s sentence pointing out the fee conflict was absent from the executed corpus.

Condition A was a basic citation request. Condition B added checking rules. Both required answers based only on the supplied documents, with document and clause IDs plus short verbatim quotations. These were typed references and quotations, not a test of a native citation feature. B added effective-date, explicit-replacement and exception checks; a rule against treating a later publication date alone as overriding; handling of unresolved equal-authority conflicts and missing information; quotation-support checks and omission of unsupported claims; and answering supported parts.

Test scope: Responses reported the model as claude-sonnet-5. The Claude command-line tool sent the documents to the model provider. Twelve questions were written in Korean and English; each was asked once per language and condition, in a new process and session (12×2×2=48). All 48 planned runs were attempted and produced 48 valid responses; there were zero invalid, skipped or retried runs. English responses had a 150-word limit including citations; Korean responses had a 600-character limit including spaces and citations. These were unequal budgets, but all outputs finished under their caps: the largest were 117 words and 371 characters, with no technical truncation. Another AI assistant, Codex, made rubric judgments against the prewritten answer key. The tested model did not grade itself, and no human scored the outputs independently.

The record describes pre-run changes from the proposed claude-sonnet-4-6, restoration of the request wording, and neutral wording for Q10. The revision timing lacks independent timestamp attestation. Exact CLI/OS versions, billing evidence and complete internal hook/memory isolation are also unverified. The separately retained experiment record contains summaries, correction history and paths to the original inputs, outputs, answer key and scoring files.

The English results come from runs using English documents and questions; they are not translated Korean results. Each cell counts questions meeting that row’s criterion out of those asked: all required facts for direct facts and applicable rules; both values and sources, an unresolved conflict and no definitive fee for conflicts; and explicitly stating that information is absent without inventing a value or turning pickup rules into return rules for missing information. Conflict and missing-information questions are shown individually.

These counts describe this single run, not a success rate or guarantee for real documents.

All four combinations correctly gave Monday–Friday, 10:00–16:00 for pickup in Q1, and the reservation code to bring in Q2. They distinguished the current fourteen-day standard loan, the seven-day rule applicable on June 15, 2026, and the three-day camera exception. These are calendar days, including weekends and holidays.

English B's Q8 response correctly said that one camera plus two standard items totals three and fits the item limit on the reference date, September 27, 2026. It omitted the frozen key’s additional explicit requirement: the maximum of one camera. The scores below therefore measure completeness against the required facts; this was not wrong arithmetic or an invented policy. The key prohibited stock or approval guarantees, and no response made one.

Across the eight questions the documents could answer (Q1–8), English A met all required facts on 8 of 8 and English B on 7 of 8. Korean A and B each met them on 8 of 8. None of the four combinations refused an answerable question: 0 of 8 each.

For the conflict questions, two notices issued on the same dates by the same body and both marked as in effect specified fees of five and eight fictional credits, with no precedence rule in the supplied collection. Every response showed both values and withheld a definitive fee. Korean B's Q9 response nevertheless omitted the shared issuer and active-status details. Handling a conflict correctly and explaining it completely are separate checks.

No response invented a late fee or treated pickup instructions as permission to return an item on Sunday. English A's Q12 response did add an inference about a Sunday due date. Counting weekends makes this plausible, but the documents do not fully specify when counting starts or whether a Sunday due date moves to another day. Codex scored the extra claim as only partly supported. The missing-information criterion still passed; that pass did not validate every added claim.

Correct answers did not always come with adequate quotations

English A's Q5 response correctly answered “seven days,” but its quotation was only “including weekends and holidays.” That passage does not contain the number seven. English B's Q8 response inserted an ellipsis into its camera-duration quotation. It preserved the duration’s meaning but failed this experiment’s verbatim-quotation criterion. Supporting a claim and reproducing the source exactly are separate checks.

Korean A’s Q3 also stated the correct replacement date but quoted only the duration sentence. English A’s Q8 supported an extra “three days” claim with a passage saying the exception remained applicable, which did not give the duration. Some A responses added correct pickup or camera facts with references but no quotations.

The columns distinguish semantic support from complete citations. B also instructed the model to omit unsupported claims, so claim selection itself could change the denominator. Answers contained different numbers of claims; these are not common-denominator performance comparisons or pooled rates. Codex assigned claim boundaries and support labels and later corrected some Q9 citation scores. The initial scores and reasons are preserved.

A document reference without a quotation did not earn full support credit on this measure. Statements that information or precedence was absent were checked separately against all six documents. Answers with no facts to cite were marked not applicable, not given automatic perfect credit.

A checklist for checking AI answers yourself

My takeaway is to compare the answer and the quoted passage side by side. These results do not establish that B is better or that an assistant should hold back an answer by default. The checklist below is an untested example format. Whether people detect more errors or check real documents more easily has not been measured.

When testing examples whose answers you already know, write down the expected answer before asking the AI. For everyday checking, you can leave that field blank and start with the source documents.

For a test, include a prewritten answer key and all four question types. Record quotation support, whether the rule applies, and any unnecessary refusal. Searching was not tested here, but my advice for tools that search documents is not to turn “the search did not find it” directly into “the documents do not contain it.”

You can try two new conversations with the same documents that are safe to share and the same question. Use the common request in both and add the complete B instructions to only one. This exercise uses the original wording; one attempt each cannot establish a winner. Its reference date and length cap belong to this experiment and need separate choices for another task.

Common request

Answer the question in English using only the six documents below. The reference date is 2026-09-27 unless the question specifies another date. Attach [document ID §clause ID] and a short verbatim quotation to factual claims. Do not use outside knowledge. Keep the answer within 150 words, including citations.

B’s additional instructions

Check effective dates, explicit replacement and the scope of exceptions. Do not treat a later publication date alone as overriding another rule. If equally authoritative documents conflict without a documented resolution, show both sources and state that the conflict prevents a definite answer. If necessary information is absent, state “Not stated in the supplied documents” without guessing. Check that each quotation actually supports its attached claim and omit claims it does not support. Answer any supported part of the question.

Providing every document was a favorable condition; 48 calls were not 48 independent real-world tasks. Both languages used the same English system instruction and one set of fictional values. Sensitivity to system language, values, length caps and order was not tested. Each question was run once per language and condition, using one model, short synthetic documents and related questions. It cannot establish language or product superiority, long-document or retrieval performance, or a general error rate. Both conditions already requested citations, and B changed a bundle of rules, so the test cannot isolate the effect of citations or of any individual rule. Human checking time was not measured. Whether the checklist saves time or improves productivity remains untested.

AI assisted with research and drafting. The author reviewed the final manuscript and directed the revisions.