1How do I know each sentence of the answer is actually backed by a source?
Citing a document is not the same as being supported by it. Self-RAG makes three explicit judgements: does this question need a lookup, is this passage relevant, and does the passage really support what was written. Only text that passes all three reaches the reader, with the passage that backs it.
You may need it if
- Answers cite a document, but the document does not say what the answer claims.
- Your answers carry risk: contract terms, compliance rules, medical or financial facts.
- Reviewers spend their time checking citations by hand.
2When are three checks per answer too many?
Use it when
- A wrong claim with a citation is worse than no answer, because readers trust citations.
- Your documents are the authority and the answer must not go beyond them.
- Some questions need no lookup at all and you do not want to search for those.
Avoid it when
- The real difficulty is finding the documents, through several reformulated searches.
Use Agentic RAG instead - Your index often has nothing useful and you need a second source to fall back on.
Use Corrective RAG instead - Users want a quick conversational reply and an occasional loose claim is acceptable.
Use Adaptive RAG instead
What it costs
- Model calls
- 1 to decide + 1 per passage graded + 1 per draft + 1 per support check (often 8 to 15)
- Latency
- high
- Complexity
- medium
3Where do the judgements happen as the answer is built?
A first judgement decides whether to retrieve. Retrieved passages are graded one by one and only relevant ones are kept. The writer drafts from them, citing each passage. A support check compares every claim with its passage; unsupported drafts are rewritten once, then the answer leaves.
- Step 1 of 5
Does this need a lookup?
The first judgement is whether to search at all. "Summarise this paragraph" does not. "What is our refund window?" does. Skipping needless searches saves cost and keeps unrelated passages out.
- Step 2 of 5
Retrieve, then judge each passage
The retriever returns a handful of passages. Each one is graded alone against the question. Only endorsed passages move on; the rest are dropped before any writing.
- Step 3 of 5
Draft with a passage behind every claim
The writer drafts from the endorsed passages and tags each claim with the passage it relies on. A claim without a passage must be removed or marked as general knowledge.
- Step 4 of 5
Check support, claim by claim
A separate check reads each claim next to its passage. Does the passage say this, in substance? Partly supported claims count as failures.
- Step 5 of 5
Rewrite once, then release
Failed claims go back to the writer once, to be corrected from the passage or dropped. The final answer keeps only supported claims, and says what it could not confirm.
What each part does
- Question
- The user's question in plain language.
- Retrieve?
- Decides whether the question depends on your documents or can be answered without them.
- Retriever
- Searches the index and returns the top passages with their sources.
- Document index
- Your reference documents, split into passages and indexed ahead of time.
- Relevant?
- Grades each passage on its own and keeps only those that bear on the question.
- Writer
- Drafts the answer from the endorsed passages, tagging each claim with the passage behind it.
- Supported?
- Checks every claim against its passage: fully supported, partly, or not at all.
- Answer
- The answer made only of supported claims, each with its citation.
- Retrieve? → Retriever: depends on documents
- Retrieve? → Writer: no lookup needed
- Retriever → Relevant?: passages
- Relevant? → Writer: endorsed passages
- Writer → Supported?: draft with citations
- Supported? → Writer: unsupported claims, one rewrite left
- Supported? → Answer: all supported, or rewrite used
Where it connects
A question in plain language. Optionally, a note on how strict the support check should be.
When: Every remaining claim is supported, or the single rewrite has been used.
The answer, each claim with its supporting passage and source, and a list of claims that were dropped as unsupported.
Passages from your reference documents with source and date. Read during the run, never written by it.
4What do I ask my assistant to set up?
The agents to launch
Writer
First say whether the question needs your documents, in one word. When writing, use only the endorsed passages. Write short claims, each followed by the passage number it relies on. When given failed claims, correct them from their passage or delete them. Never cite a passage for something it does not say.
Sees: The question, the endorsed passages and, on a rewrite, the list of failed claims. Not the rejected passages.
Judge, One call per passage, then one call per claim, run in parallel
You make one judgement per call. For relevance, given a question and one passage, answer relevant or irrelevant with the deciding sentence. For support, given one claim and one passage, answer supported, partly or unsupported, quoting the passage.
Sees: One question and one passage, or one claim and one passage. Never the full draft, the other passages or the writer's reasoning.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Self-RAG" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Self-RAG Judge every step, and answer only from passages that hold up. Input: the task above. It must provide: A question in plain language. Optionally, a note on how strict the support check should be. If something essential is missing, ask for it before starting. Agents to launch: - Writer: First say whether the question needs your documents, in one word. When writing, use only the endorsed passages. Write short claims, each followed by the passage number it relies on. When given failed claims, correct them from their passage or delete them. Never cite a passage for something it does not say. Context for this agent: The question, the endorsed passages and, on a rewrite, the list of failed claims. Not the rejected passages. - Judge (One call per passage, then one call per claim, run in parallel): You make one judgement per call. For relevance, given a question and one passage, answer relevant or irrelevant with the deciding sentence. For support, given one claim and one passage, answer supported, partly or unsupported, quoting the passage. Context for this agent: One question and one passage, or one claim and one passage. Never the full draft, the other passages or the writer's reasoning. Procedure: 1. Confirm the documents are indexed as passages with their source. Build the index once, outside the run. 2. Ask the writer whether the question needs a lookup. If not, have it answer directly and mark the answer as not based on your documents. 3. Otherwise search the index and keep the top five passages. Launch one judge call per passage for relevance. 4. If no passage is endorsed, stop and report that the documents do not cover the question. 5. Launch the writer with the endorsed passages. Split its draft into claims and launch one judge call per claim for support. 6. Send any claim judged partly or unsupported back to the writer once. Drop any claim that still fails. Stop rule: Stop after the support check that follows at most one rewrite. Claims still unsupported are removed, never released. Exit: - `answer`, when: Every remaining claim is supported, or the single rewrite has been used. Deliver: The answer, each claim with its supporting passage and source, and a list of claims that were dropped as unsupported. # Final deliverable The answer built from supported claims, each with its passage and source, then the list of claims dropped as unsupported and the passages rejected as irrelevant.
The same thing in code
import asyncio
import operator
from typing import Annotated, Literal, TypedDict
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel
llm = init_chat_model("anthropic:claude-sonnet-5-5")
WRITER = ("Use only the endorsed passages. Write short claims, one fact each, each tagged with the number of the "
"passage it relies on. When given failed claims, correct them from their passage or delete them. "
"Never cite a passage for something it does not say.")
JUDGE = ("You make one judgement per call. For relevance, given a question and one passage, answer relevant or "
"irrelevant with the deciding sentence. For support, given one claim and one passage, answer supported, "
"partly or unsupported, quoting the passage.")
class Need(BaseModel):
needs_documents: bool
class Relevance(BaseModel):
relevant: bool
sentence: str
class Claim(BaseModel):
text: str
passage: int
class Draft(BaseModel):
claims: list[Claim]
class Support(BaseModel):
verdict: Literal["supported", "partly", "unsupported"]
quote: str
class Input(TypedDict):
task: str
class State(Input):
needs: bool
direct: str
endorsed: list[str]
rejected: list[str]
pending: list[Claim]
kept: Annotated[list[Claim], operator.add]
failed: list[Claim]
rewrites: int
def search_index(query: str, k: int = 5) -> list[str]:
"""Stub: search your passage index; return each passage with its source."""
return [f"(guide.pdf) passage about {query}"][:k]
async def ask[T: BaseModel](schema: type[T], system: str, user: str) -> T:
return schema.model_validate(await llm.with_structured_output(schema).ainvoke([("system", system), ("user", user)]))
async def need(state: State) -> dict:
r = await ask(Need, WRITER, f"Does this question need the documents? Question: {state['task']}")
return {"needs": r.needs_documents}
async def direct(state: State) -> dict:
return {"direct": (await llm.ainvoke(state["task"])).text}
async def relevance(state: State) -> dict:
# one judge call per passage: one question and one passage, nothing else
passages = search_index(state["task"])
grades = await asyncio.gather(*(ask(Relevance, JUDGE, f"Question: {state['task']}\nPassage: {p}") for p in passages))
return {"endorsed": [p for p, g in zip(passages, grades) if g.relevant],
"rejected": [p for p, g in zip(passages, grades) if not g.relevant], "rewrites": 0}
async def write(state: State) -> dict:
# the writer sees the endorsed passages and, on a rewrite, the failed claims; never the rejected ones
numbered = "\n".join(f"[{i}] {p}" for i, p in enumerate(state["endorsed"]))
failed = "\n".join(f"- {c.text} [{c.passage}]" for c in state.get("failed", []))
ask_for = f"Correct or delete these failed claims:\n{failed}" if failed else "Draft the answer."
draft = await ask(Draft, WRITER, f"Question: {state['task']}\nEndorsed passages:\n{numbered}\n{ask_for}")
return {"pending": draft.claims, "rewrites": state["rewrites"] + (1 if failed else 0)}
async def support(state: State) -> dict:
# one judge call per claim, with only that claim and its passage
ok = [c for c in state["pending"] if 0 <= c.passage < len(state["endorsed"])]
checks = await asyncio.gather(*(ask(Support, JUDGE, f"Claim: {c.text}\nPassage: {state['endorsed'][c.passage]}") for c in ok))
return {"kept": [c for c, s in zip(ok, checks) if s.verdict == "supported"],
"failed": [c for c, s in zip(ok, checks) if s.verdict != "supported"]}
builder = StateGraph(State, input_schema=Input)
for name, fn in [("need", need), ("direct", direct), ("relevance", relevance), ("write", write), ("support", support)]:
builder.add_node(name, fn)
builder.add_edge(START, "need")
builder.add_conditional_edges("need", lambda s: "relevance" if s["needs"] else "direct", ["relevance", "direct"])
builder.add_conditional_edges("relevance", lambda s: "write" if s["endorsed"] else END, ["write", END])
builder.add_edge("write", "support")
# the gate: one rewrite at most; claims that still fail are dropped
builder.add_conditional_edges("support", lambda s: "write" if s["failed"] and s["rewrites"] == 0 else END, ["write", END])
builder.add_edge("direct", END)
graph = builder.compile()
async def run(task: str):
s = await graph.ainvoke({"task": task})
if not s["needs"]:
return {"answer": s["direct"], "based_on_documents": False}
if not s["endorsed"]:
return {"answer": "The documents do not cover this question.", "rejected": s["rejected"]}
return {"claims": [(c.text, s["endorsed"][c.passage]) for c in s["kept"]],
"dropped": [c.text for c in s.get("failed", [])], "rejected": s["rejected"]}
Replace search_index with your passage index, and check that the writer keeps one fact per claim, or partly supported claims slip through.
from typing import Any
from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
AGENTS = {
"writer": AgentDefinition(
description="Decides whether a lookup is needed, then writes short cited claims from endorsed passages.",
prompt=("First say whether the question needs the documents. When writing, use only the endorsed passages: "
"short claims, one fact each, each tagged with the passage number it relies on. When given failed "
"claims, correct them from their passage or delete them. Never cite a passage for something it does not say."),
tools=[]),
"judge": AgentDefinition(
description="Makes one relevance or support judgement on one passage.",
prompt=("You make one judgement per call. For relevance, given a question and one passage, answer relevant or "
"irrelevant with the deciding sentence. For support, given one claim and one passage, answer "
"supported, partly or unsupported, quoting the passage."),
tools=[]),
}
def obj(**props: Any) -> dict[str, Any]:
return {"type": "object", "required": list(props), "properties": props}
NEED = obj(needs_documents={"type": "boolean"})
VERDICTS = obj(verdicts={"type": "array", "items": {"type": "string"}})
CLAIMS = obj(claims={"type": "array", "items": obj(text={"type": "string"}, passage={"type": "integer"})})
def search_index(query: str, k: int = 5) -> list[str]:
"""Stub: search your passage index; return each passage with its source."""
return [f"(guide.pdf) passage about {query}"][:k]
async def ask(prompt: str, schema: dict[str, Any] | None = None) -> Any:
fmt = {"type": "json_schema", "schema": schema} if schema else None
options = ClaudeAgentOptions(agents=AGENTS, tools=["Agent"], allowed_tools=["Agent"], output_format=fmt)
async for message in query(prompt=prompt, options=options):
if isinstance(message, ResultMessage):
return message.structured_output if schema else message.result
raise RuntimeError("no result")
async def judge_each(items: list[str], kind: str) -> list[str]:
# one judge sub-agent per item, in parallel; each sees one pair, never the full draft
listing = "\n".join(f"{i}. {x}" for i, x in enumerate(items))
out = await ask(f"Launch one judge sub-agent per item below, in parallel, for a {kind} judgement. Give each only "
f"its own item.\n{listing}\nReturn each verdict word in item order.", VERDICTS)
return out["verdicts"]
async def check(claims: list[dict[str, Any]], endorsed: list[str]) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
claims = [c for c in claims if 0 <= c["passage"] < len(endorsed)]
verdicts = await judge_each([f"Claim: {c['text']}\nPassage: {endorsed[c['passage']]}" for c in claims], "support")
return ([c for c, v in zip(claims, verdicts) if v == "supported"],
[c for c, v in zip(claims, verdicts) if v != "supported"])
async def run(task: str):
need = await ask(f"Ask the writer sub-agent whether this question needs the documents: {task}", NEED)
if not need["needs_documents"]:
answer = await ask(f"Have the writer sub-agent answer directly: {task}\nReturn its answer verbatim.")
return {"answer": answer, "based_on_documents": False}
passages = search_index(task)
grades = await judge_each([f"Question: {task}\nPassage: {p}" for p in passages], "relevance")
endorsed = [p for p, g in zip(passages, grades) if g == "relevant"]
rejected = [p for p, g in zip(passages, grades) if g != "relevant"]
if not endorsed:
return {"answer": "The documents do not cover this question.", "rejected": rejected}
numbered = "\n".join(f"[{i}] {p}" for i, p in enumerate(endorsed))
# the writer sees the endorsed passages only, never the rejected ones
draft = await ask(f"Use the writer sub-agent. Give it only:\nQuestion: {task}\nEndorsed passages:\n{numbered}\n"
"Return its claims.", CLAIMS)
kept, failed = await check(draft["claims"], endorsed)
if failed: # the gate: one rewrite, then anything still failing is dropped
listing = "\n".join(f"- {c['text']} [{c['passage']}]" for c in failed)
fixed = await ask(f"Use the writer sub-agent. Give it only:\nEndorsed passages:\n{numbered}\n"
f"Correct or delete these failed claims:\n{listing}\nReturn its claims.", CLAIMS)
more, failed = await check(fixed["claims"], endorsed)
kept += more
return {"claims": [(c["text"], endorsed[c["passage"]]) for c in kept],
"dropped": [c["text"] for c in failed], "rejected": rejected}
Replace search_index with your passage index, and check that the writer keeps one fact per claim, or partly supported claims slip through.
import asyncio
from typing import Literal
from agents import Agent, Runner
from pydantic import BaseModel
class Need(BaseModel):
needs_documents: bool
class Claim(BaseModel):
text: str
passage: int
class Draft(BaseModel):
claims: list[Claim]
class Relevance(BaseModel):
relevant: bool
sentence: str
class Support(BaseModel):
verdict: Literal["supported", "partly", "unsupported"]
quote: str
writer = Agent(
name="Writer",
instructions=("First say whether the question needs the documents. When writing, use only the endorsed passages: "
"short claims, one fact each, each tagged with the passage number it relies on. When given failed "
"claims, correct them from their passage or delete them. Never cite a passage for something it does not say."))
judge = Agent(
name="Judge",
instructions=("You make one judgement per call. For relevance, given a question and one passage, answer relevant or "
"irrelevant with the deciding sentence. For support, given one claim and one passage, answer "
"supported, partly or unsupported, quoting the passage."))
# same briefs, one structured output per kind of judgement
need_writer, draft_writer = writer.clone(output_type=Need), writer.clone(output_type=Draft)
relevance_judge, support_judge = judge.clone(output_type=Relevance), judge.clone(output_type=Support)
def search_index(query: str, k: int = 5) -> list[str]:
"""Stub: search your passage index; return each passage with its source."""
return [f"(guide.pdf) passage about {query}"][:k]
async def check(claims: list[Claim], endorsed: list[str]) -> tuple[list[Claim], list[Claim]]:
# one judge run per claim, with only that claim and its passage
claims = [c for c in claims if 0 <= c.passage < len(endorsed)]
runs = await asyncio.gather(*(Runner.run(support_judge, f"Claim: {c.text}\nPassage: {endorsed[c.passage]}") for c in claims))
ok = [r.final_output_as(Support).verdict == "supported" for r in runs]
return [c for c, k in zip(claims, ok) if k], [c for c, k in zip(claims, ok) if not k]
async def run(task: str):
need = await Runner.run(need_writer, f"Does this question need the documents? {task}")
if not need.final_output_as(Need).needs_documents:
direct = await Runner.run(writer, task)
return {"answer": str(direct.final_output), "based_on_documents": False}
passages = search_index(task)
# one judge run per passage: one question and one passage, nothing else
runs = await asyncio.gather(*(Runner.run(relevance_judge, f"Question: {task}\nPassage: {p}") for p in passages))
grades = [r.final_output_as(Relevance).relevant for r in runs]
endorsed = [p for p, g in zip(passages, grades) if g]
rejected = [p for p, g in zip(passages, grades) if not g]
if not endorsed:
return {"answer": "The documents do not cover this question.", "rejected": rejected}
numbered = "\n".join(f"[{i}] {p}" for i, p in enumerate(endorsed))
draft = await Runner.run(draft_writer, f"Question: {task}\nEndorsed passages:\n{numbered}")
kept, failed = await check(draft.final_output_as(Draft).claims, endorsed)
if failed: # the gate: one rewrite, then anything still failing is dropped
listing = "\n".join(f"- {c.text} [{c.passage}]" for c in failed)
fixed = await Runner.run(draft_writer, f"Endorsed passages:\n{numbered}\nCorrect or delete these failed claims:\n{listing}")
more, failed = await check(fixed.final_output_as(Draft).claims, endorsed)
kept += more
return {"claims": [(c.text, endorsed[c.passage]) for c in kept],
"dropped": [c.text for c in failed], "rejected": rejected}
Replace search_index with your passage index, and check that the writer keeps one fact per claim, or partly supported claims slip through.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
const model = anthropic('claude-sonnet-5-5');
const WRITER =
'First say whether the question needs the documents. When writing, use only the endorsed passages: short ' +
'claims, one fact each, each tagged with the passage number it relies on. When given failed claims, correct ' +
'them from their passage or delete them. Never cite a passage for something it does not say.';
const JUDGE =
'You make one judgement per call. For relevance, given a question and one passage, answer relevant or ' +
'irrelevant with the deciding sentence. For support, given one claim and one passage, answer supported, ' +
'partly or unsupported, quoting the passage.';
const draftSchema = z.object({ claims: z.array(z.object({ text: z.string(), passage: z.number().int() })) });
type Claim = z.infer<typeof draftSchema>['claims'][number];
// Stub: search your passage index; return each passage with its source.
async function searchIndex(query: string, k = 5): Promise<string[]> {
return [`(guide.pdf) passage about ${query}`].slice(0, k);
}
async function ask<T>(instructions: string, prompt: string, schema: z.ZodType<T>): Promise<T> {
return (await generateText({ model, instructions, prompt, output: Output.object({ schema }) })).output;
}
async function check(claims: Claim[], endorsed: string[]): Promise<[Claim[], Claim[]]> {
// one judge call per claim, with only that claim and its passage
const valid = claims.filter((c) => c.passage >= 0 && c.passage < endorsed.length);
const verdicts = await Promise.all(valid.map((c) => ask(JUDGE, `Claim: ${c.text}\nPassage: ${endorsed[c.passage]}`,
z.object({ verdict: z.enum(['supported', 'partly', 'unsupported']), quote: z.string() }))));
return [valid.filter((_, i) => verdicts[i].verdict === 'supported'), valid.filter((_, i) => verdicts[i].verdict !== 'supported')];
}
export async function run(task: string) {
const need = await ask(WRITER, `Does this question need the documents? ${task}`, z.object({ needsDocuments: z.boolean() }));
if (!need.needsDocuments) {
const { text } = await generateText({ model, instructions: WRITER, prompt: task });
return { answer: text, basedOnDocuments: false };
}
const passages = await searchIndex(task);
// one judge call per passage: one question and one passage, nothing else
const grades = await Promise.all(passages.map((p) => ask(JUDGE, `Question: ${task}\nPassage: ${p}`,
z.object({ relevant: z.boolean(), sentence: z.string() }))));
const endorsed = passages.filter((_, i) => grades[i].relevant);
const rejected = passages.filter((_, i) => !grades[i].relevant);
if (!endorsed.length) return { answer: 'The documents do not cover this question.', rejected };
const numbered = endorsed.map((p, i) => `[${i}] ${p}`).join('\n');
const draft = await ask(WRITER, `Question: ${task}\nEndorsed passages:\n${numbered}`, draftSchema);
let [kept, failed] = await check(draft.claims, endorsed);
if (failed.length) { // the gate: one rewrite, then anything still failing is dropped
const listing = failed.map((c) => `- ${c.text} [${c.passage}]`).join('\n');
const fixed = await ask(WRITER, `Endorsed passages:\n${numbered}\nCorrect or delete these failed claims:\n${listing}`, draftSchema);
const [more, still] = await check(fixed.claims, endorsed);
kept = kept.concat(more);
failed = still;
}
return {
claims: kept.map((c) => ({ claim: c.text, passage: endorsed[c.passage] })),
dropped: failed.map((c) => c.text),
rejected,
};
}
Replace searchIndex with your passage index, and check that the writer keeps one fact per claim, or partly supported claims slip through.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- Long claims mix supported and unsupported parts and pass as "partly". Ask the writer for one fact per claim.
- A judge that sees the whole draft grades the story, not the claim. Give it one claim and one passage at a time.
- Dropping failed claims can leave the main question unanswered. Say so in the answer instead of releasing a hollow one.
Chain it
Its result can feed
- Reflection conditional
Condition: The link applies when A's answer is a prose answer handed over as the result to improve, not a short fact or label; the evidence list goes along only as reference.
- Chain of Verification conditional
Condition: The link applies to the answer part of A's payload, handed over as an existing answer whose factual claims are to be checked; the claim-passage evidence is not part of the work item.
- Debate conditional
Condition: The link applies when the original task asks for a judgement, interpretation or contested conclusion rather than a plain fact to look up, and only the answer is passed as the position to challenge.
- Constitutional Critique conditional
Condition: The link applies to the answer text of A's payload, submitted as the text to check; the evidence list is not part of the checked text.
- Textual Backpropagation conditional
Condition: The link applies to the answer part of A's payload only; the claim-by-passage evidence list and dropped-claims list are not a kind B's entry accepts.