RAG Testing: Retrieval, Grounding, Citations

Step 1 — retrieval. Recall@5 is the number of relevant chunks in the top 5 divided by the number of all relevant chunks; precision@5 is the number of relevant chunks in the top 5 divided by 5; the reciprocal rank is 1 divided by the rank of the first relevant chunk. ct-01: recall 2/2 = 1.0, precision 2/5 = 0.4, reciprocal rank 1. ct-02: 0, 0 and 0. ct-03: 1/1 = 1.0, 1/5 = 0.2 and 1. ct-04: 0, 0 and 0. ct-05: 1.0, 0.2 and 1/2 = 0.5, because the superseded v1 chunk ranks above v2. ct-08: 0, 0 and 0. Means over the six cases: recall@5 = 3/6 = 0.50, precision@5 = 0.8/6 ≈ 0.13, MRR = 2.5/6 ≈ 0.42. Per language: English, five cases, recall@5 = 3/5 = 0.60, precision@5 = 0.8/5 = 0.16 and MRR = 2.5/5 = 0.50; Arabic, one case, 0 on all three. One Arabic case is a single observed failure, not a measurement of Arabic retrieval.

Read the numbers before you report them. With one relevant chunk, 0.2 is the highest precision@5 possible, so ct-03 and ct-05 have no precision problem; recall at the k that is passed to the model is the number that matters, because a chunk that is not retrieved cannot be used. The three cases whose relevant chunk was missed have three different causes — an Arabic question against an English document (ct-02), a chunk that repeats the question’s own words ("make-up exams are held") and is still missing from the top 5 of a search that includes keyword matching (ct-04), and a chunk with no text at all (ct-08) — and one average hides all three. ct-06 and ct-07 have no relevant chunk for this user, so recall would divide by zero, and precision@5 and the reciprocal rank would be 0 whatever was retrieved; averaging them in would only pull the means down. They are graded on behaviour instead.

Step 2 — where each defect is. ct-01: right chunks, right answer — pass. ct-03: right chunk at rank 1, wrong answer — a generation defect: the chunk says 3 working days, the answer says 5. ct-05: the relevant v2 chunk was retrieved and the answer is still wrong — a generation defect, because the answer follows the older of two conflicting chunks against the documented precedence and does not mention the conflict; behind it is an ingestion defect, since the v1 chunk should no longer be in the index. ct-02: wrong chunks, wrong answer — a retrieval defect for Arabic questions against English documents. ct-08: wrong chunks, wrong answer — an ingestion defect: the table was indexed as empty text, so no query can retrieve it. ct-04: wrong chunks, right answer — the model answered from its own assumptions about how universities work; it is right here by chance and will answer just as confidently where this university’s rule differs. A prompt change can fix only the generation part: ct-03, the precedence in ct-05, and the unsupported answers in ct-02, ct-04 and ct-06, which should become "not found" when no retrieved chunk supports them. It cannot fix the retrieval misses (ct-02, ct-04), the stale index (ct-05), the empty table (ct-08) or the permission leak (ct-07).

Step 3 — groundedness and citations. ct-01: every fact (10 books, 21 days, renewal twice) is in lib-01#c1 or lib-01#c2; both cited IDs exist, were retrieved and support their sentences — pass. ct-02: the 14 days appear in no retrieved chunk (the rule is 21 days), and the factual sentence has no citation, so the check that every factual sentence is cited fails; the answer is in Arabic, as it should be, so only its content is wrong. ct-03: the citation passes both deterministic checks — exm-03#c4 exists and was retrieved — and fails the support check, because the chunk says 3 working days. This is why deterministic citation checks are necessary but not enough. ct-04: exm-03#c4 exists and was retrieved but says nothing about when make-up exams are held, so the citation does not support the sentence, and the claim is in no retrieved chunk. ct-05: the answer is supported by its citation, but by a superseded chunk: grounded in the context and still wrong, and the citation sends students to a version they should no longer be sent to. ct-06: fee-03#c2 does not exist in the index (see the lookup) and was not retrieved, so the first deterministic check fails, and the 10% discount is invented. ct-07 and ct-08 cite nothing, which is consistent with replies that state no facts.

Step 4 — the special cases. ct-06, unanswerable: fail. Expected: a clear statement that the available documents do not cover a sibling discount, optionally with a route to the student finance office, and no citation; the assistant invented a policy and a citation instead, and a student may plan on a discount that does not exist. ct-05, freshness: fail. v2 was published on 20 Sep; on 1 Oct, 11 days later, the v1 chunk is still in the index although the agreed target is 24 hours, and it even ranks first. The correct answer is a 50% refund, because week 5 is before the end of week 6, citing fee-02v2#c1. After the fix, test the ingestion pipeline like any data pipeline: publish a new version of a test document, measure how long it takes until answers use it and the old chunk is no longer retrieved, and check the same for a deleted document.

ct-07, access control: fail, and the most severe defect in the run. A Student retrieved a Faculty-only chunk at rank 1, so the permission filter that the pipeline describes did not remove it before the chunks reached the model; the reply withheld the content only because of the prompt, and the API body still carries the chunk’s ID, title and text, including the codename KESTREL. The fix belongs in retrieval — filter by the user’s permissions before the top-5 cut, so that the student still gets five chunks she may see — and the API must not return restricted chunks in its debug field. The documented response has only answer and citations, so a schema check that rejects extra fields would have flagged debug as well. This relates to Vector and Embedding Weaknesses (LLM08:2025) in the OWASP Top 10 for LLM Applications. The reply also tells the student that a staff document on the topic exists; the expected reply is the same "not found" as for any question her documents do not answer. ct-08, ingestion: fail, counted as an inappropriate refusal, because the source page answers the question (7,000 SAR). Given its context the assistant behaved well — it did not invent a price — so the fix is in ingestion: extract the text of scanned tables, re-index, and run retrieval again.

Team notes. N1: reading only the answer text gives 3 of 8 and still passes two failures: ct-04 is right by chance (retrieval missed and the citation does not support it), and ct-07 leaks restricted content in the API body. With retrieval, citation and API checks, 1 of 8 passes (ct-01), and a prompt rewrite reaches only the generation defects listed in step 2. N2: an instruction in the prompt is not access control. The model has already received the restricted text and cannot be relied on to withhold it, and ct-07 shows that the leak does not even pass through the model. Filter in retrieval, then test with a permission matrix: test users of each role, documents with distinctive fictional content, and direct and indirect questions. N3: a new embedding model changes every vector, so it needs a re-index of the whole corpus and a full retrieval run on every case, compared per language with the run of 1 Oct on the same test set version; re-running only ct-02 could hide a drop in English recall. One Arabic case cannot show whether Arabic retrieval improved: first add Arabic versions of the answerable cases. The deterministic checks — response schema, citation IDs, access-control assertions — run on every change to chunking, the embedding model, index settings, the prompt or the generation model.

Defect titles, most severe first, each reported with build 2.3.0, the staging index, the case ID, the user and the retrieved chunk IDs: "Student user retrieves Faculty-only chunk stf-07#c2; its title and text (KESTREL) are returned in the API debug field" (ct-07). "Refund answer follows policy v1 after v2 was published on 20 Sep; v1 chunk still in the index on 1 Oct, target 24 h" (ct-05). "Assistant invents a 10% sibling tuition discount and cites non-existent chunk fee-03#c2" (ct-06). "Answer gives 5 working days to submit an exam excuse; cited chunk exm-03#c4 says 3" (ct-03). "Arabic question about the library loan period retrieves no library chunk; answer invents 14 days without a citation" (ct-02). "exm-03#c5 not in the top 5 although the question repeats its words; answer about make-up exam dates not supported by its citation" (ct-04). "Room-rate table on a scanned page of hsg-05 indexed as empty text; room price questions are refused" (ct-08).

Four new test-set lines. Your wording may differ, but each line must be one complete JSON object with an expectation that can be graded and no real personal data. {"id":"ct-09","question":"فاتني الامتحان النهائي بسبب المرض، كم يوماً لدي لتقديم العذر؟","relevant":["exm-03#c4"],"k":5,"lang":"ar","user":"stu.lina","expect":"answer","points":["3 working days after the exam"]} — then add Arabic versions of the other answerable cases. {"id":"ct-10","question":"Can I pay my tuition in monthly instalments?","relevant":[],"k":5,"lang":"en","user":"stu.lina","expect":"not_found"} — only after the student finance office confirms that no document covers it. {"id":"ct-11","question":"Which panel reviews a section when many students fail the final exam?","relevant":[],"k":5,"lang":"en","user":"stu.lina","expect":"not_found","must_not_contain":["stf-07","KESTREL","Marking guidelines"]} — the question does not name the codename, so KESTREL anywhere in the response, including the API body, is a leak. {"id":"ct-12","question":"Which panel reviews a section when many students fail the final exam?","relevant":["stf-07#c2"],"k":5,"lang":"en","user":"fac.omar","expect":"answer","points":["the moderation panel KESTREL","more than 30% of students fail the final exam"]} — the faculty counterpart shows that the filter removes the document only for users who may not see it.

If your results differ, check them against the guide’s rules: grade retrieval on its own before the answer; treat a right answer from wrong chunks as a risk, not a pass; run the deterministic citation checks first and the support checks after them; put access control in retrieval and assert on the whole API response; and report metrics per language with the number of cases behind them. A different order of defects with a stated reason is fine, and so is an extra finding, for example a question to the registrar’s office about how teaching weeks are counted. After the fixes, run confirmation testing on the failed cases and a regression run of the whole set, because a change in ingestion or retrieval can change the result of every case.