Retrieval-Augmented Generation: Answering from Selected Evidence

A language model's training cannot contain every private document or every policy change that happened yesterday.

Retrieval-augmented generation, usually called RAG, supplies selected evidence at request time.

RAG retrieves relevant source material and places it in the model's context before generation.

This can improve freshness and traceability, but only when the ingestion, retrieval, permissions, prompt, and answer validation all work together.

A concrete example: annual leave policy

An employee asks, “Can unused leave move into next year?”

The system:

1. identifies the employee's country and policy access,
2. searches current approved policy chunks,
3. retrieves the carry-over rule and its exceptions,
4. places those passages in the prompt,
5. asks the model to answer only from them,
6. and returns citations to the source sections.

The model did not permanently learn the policy. It used retrieved evidence for this answer.

The RAG pipeline

A typical RAG system has two broad paths.

The ingestion path:

• collects sources,
• parses content,
• divides it into chunks,
• attaches metadata,
• creates searchable indexes,
• and updates or deletes them.

The request path:

• interprets the question,
• retrieves candidates,
• reranks or filters them,
• builds context,
• generates an answer,
• and records evidence and feedback.

Source quality comes first

Retrieval cannot repair an ungoverned knowledge base.

Decide which sources are authoritative, who owns them, when they expire, and how conflicting versions are resolved. Remove drafts, duplicated pages, obsolete procedures, and navigation noise.

When two valid sources disagree, the application needs precedence rules or should expose the conflict rather than inventing a compromise.

Parsing and chunking

Documents are not plain uninterrupted text. They contain:

• headings,
• tables,
• lists,
• footnotes,
• images,
• page headers,
• and cross-references.

Parsing should preserve useful structure. Chunking should keep a rule with its conditions and exceptions. Store section paths and page references so answers can cite material users can inspect.

Query understanding

The raw user wording may not be the best search query.

The application can:

• normalize acronyms,
• add known product or region context,
• split a multi-part question,
• generate alternate formulations,
• or route exact identifiers to keyword search.

Query rewriting must not silently change intent. Keep the original request and evaluate transformations on real examples.

Retrieval and reranking

Initial retrieval may combine:

• semantic vector search,
• keyword search,
• metadata filters,
• and structured database lookup.

A reranker can then score a smaller candidate set more carefully against the question. This two-stage design balances speed and relevance.

Retrieve enough evidence to cover the task without flooding the context with loosely related text.

Permissions before relevance

The most relevant document is not necessarily one the user may see.

Carry identity and authorization into retrieval. Apply tenant, role, document, region, and sensitivity controls before content reaches the model. The generated answer must not reveal facts from a source merely because the source itself is hidden.

Recheck permissions when cached retrieval results are reused.

Building the model context

Clearly separate:

• trusted application instructions,
• the user's question,
• retrieved source passages,
• and required output format.

Label every passage with a stable source identifier. Tell the model how to handle missing, conflicting, or insufficient evidence. Ordering can matter, so avoid always placing weak evidence first.

Citations are claims, not decoration

A citation should support the sentence attached to it.

The model may cite a retrieved passage that is related but does not actually prove the claim. Validate citation identifiers, restrict citations to supplied sources, and evaluate entailment between claims and evidence.

For sensitive applications, extract claims and verify each against the cited text before returning the answer.

RAG does not eliminate hallucinations

The model can:

• ignore the best passage,
• combine incompatible sources,
• misread a table,
• overgeneralize an exception,
• or add unsupported details.

RAG changes the evidence available; it does not make generation deterministic or truthful. Applications need explicit no-answer behavior and thresholds for human escalation.

Prompt injection in documents

A retrieved page may contain “ignore previous instructions” or malicious tool directions.

Treat source content as untrusted data. Do not grant it instruction authority. Restrict tools, isolate quoted passages, sanitize unsupported formats, and test adversarial documents. Permissions and action controls must remain outside the model's discretion.

Freshness and deletion

Track source version, effective date, ingestion time, and index state.

When a policy changes, the old chunks must be retired promptly. When access is revoked or data is deleted, remove derived vectors, cached passages, generated summaries, and other copies according to policy.

Freshness service objectives make stale knowledge measurable.

Evaluate each stage

Use a representative question set and measure:

• whether the needed source was ingested,
• retrieval recall,
• ranking quality,
• answer correctness,
• citation support,
• no-answer behavior,
• permission enforcement,
• latency,
• and cost.

Stage-level measures explain failure. If the right passage was never retrieved, changing the generation prompt alone will not fix the system.

Production feedback

Log the question category, retrieved source IDs, versions, answer, citations, latency, and user outcome with suitable privacy controls.

Review:

• unanswered questions,
• unsupported claims,
• low-quality sources,
• frequent reformulations,
• and successful escalations.

Use that evidence to improve source content, chunking, retrieval, prompts, and product workflow.

When not to use RAG

Use deterministic code or structured queries when the task requires:

• exact account balances,
• arithmetic,
• transaction state,
• permission decisions,
• or complete enumeration.

RAG is useful for finding and explaining unstructured evidence. It should not replace authoritative computation.

Knowledge check

1. What does RAG add at request time?
2. Why is source governance part of RAG quality?
3. How do retrieval and reranking differ?
4. Why do citations require validation?
5. Which failures remain possible after relevant evidence is retrieved?

The one idea to remember

RAG grounds generation in selected current or private evidence, but reliability comes from the entire pipeline: authoritative sources, faithful parsing, useful chunks, secure retrieval, supported citations, explicit uncertainty, continuous evaluation, and timely deletion.