1What happens when the search returns the wrong documents and the model answers anyway?
A model given off-topic documents still writes a fluent answer, built on them or on guesses. Corrective RAG adds a grader between search and answer. It judges each document's relevance. When too little survives, the query is rewritten and a second source, such as web search, is tried before anything is written.
You may need it if
- Answers quote real documents that have nothing to do with the question.
- Your index covers your product well but customers ask about things outside it, such as third-party integrations.
- You cannot tell from the answer whether the retrieved documents were any good.
2When is grading every document overkill?
Use it when
- Your search returns a mix of relevant and irrelevant documents.
- A backup source exists for questions your index does not cover, such as the public web or a partner's documentation.
- An answer built on the wrong document costs more than one extra model call per document.
Avoid it when
- The first search rarely contains the answer and the system must explore with several different queries.
Use Agentic RAG instead - The documents are fine but you need proof that each sentence of the answer is supported by them.
Use Self-RAG instead - You must never use sources outside your own documents, and the index already returns precise results.
What it costs
- Model calls
- 1 per document graded + 1 to write, + 1 rewrite and more grading when the fallback runs
- Latency
- medium
- Complexity
- medium
3How do documents get checked before they reach the answer?
The retriever pulls documents from the index. The grader marks each one relevant or not. If enough relevant evidence remains, the writer answers from it. If not, the query is rewritten for a second source, whose results are graded once, then the writer answers from what passed.
- Step 1 of 5
A normal search
The retriever searches your index with the question, as any retrieval system would. Five documents come back. Nothing has been checked yet.
- Step 2 of 5
Each document is graded
The grader looks at one document at a time, with the question, and answers one thing: does this document help answer it? Irrelevant documents are dropped here.
- Step 3 of 5
Enough to go on?
A plain rule counts what survived. With at least two relevant documents, the system goes straight to writing. With fewer, it corrects course instead.
- Step 4 of 5
Correct course once
The rewriter turns the question into a query suited to the second source, such as web search. Those results go back to the same grader. This happens at most once.
- Step 5 of 5
Answer from what passed
The writer sees only the documents that passed grading. Each claim cites its source, and sources from the fallback are marked as external.
What each part does
- Question
- The user's question in plain language.
- Retriever
- Searches the index with the question and returns the top documents.
- Document index
- Your own documents, indexed for search ahead of time.
- Grader
- Judges each document separately: does it contain information that answers this question?
- Evidence good?
- Counts the relevant documents and checks whether the fallback was already used.
- Query rewriter
- Turns the question into a search query suited to the second source.
- Second source
- Web search or another collection, used only when the index fell short.
- Writer
- Answers from the documents that passed grading, citing them.
- Answer
- The answer with its sources, marked with where each source came from.
- Retriever → Grader: documents
- Grader → Evidence good?: relevance per document
- Evidence good? → Writer: enough relevant, or fallback used
- Evidence good? → Query rewriter: too little, fallback unused
- Query rewriter → Second source: rewritten query
- Second source → Grader: new documents to grade
Where it connects
A question in plain language. Optionally, the list of outside sources the system is allowed to use.
When: Enough relevant documents were found, or the fallback has run once.
The answer, the documents that passed grading with their sources, whether the fallback was used, and a note when evidence remained thin.
Your documents with their source and date, searchable by query. Read during the run, never written by it.
4What do I ask my assistant to set up?
The agents to launch
Grader, One call per document, run in parallel
You receive one question and one document. Answer "relevant" only if the document contains information that helps answer this specific question, and quote the sentence that does. Otherwise answer "irrelevant". Do not answer the question.
Sees: The question and a single document. Never the other documents or the grades given so far.
Query rewriter
The documents found did not answer the question. Write one search query for the second source, using the terms that source is likely to use. Keep the meaning of the question exactly.
Sees: The question and the name of the second source. Not the rejected documents.
Writer
Answer the question using only the documents provided. Cite the source of each claim and mark external sources. If the documents leave part of the question open, say so.
Sees: The question and the documents that passed grading. Not the rejected documents or the grades.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Corrective RAG" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Corrective RAG Grade what was retrieved, and fetch elsewhere when it falls short. Input: the task above. It must provide: A question in plain language. Optionally, the list of outside sources the system is allowed to use. If something essential is missing, ask for it before starting. Agents to launch: - Grader (One call per document, run in parallel): You receive one question and one document. Answer "relevant" only if the document contains information that helps answer this specific question, and quote the sentence that does. Otherwise answer "irrelevant". Do not answer the question. Context for this agent: The question and a single document. Never the other documents or the grades given so far. - Query rewriter: The documents found did not answer the question. Write one search query for the second source, using the terms that source is likely to use. Keep the meaning of the question exactly. Context for this agent: The question and the name of the second source. Not the rejected documents. - Writer: Answer the question using only the documents provided. Cite the source of each claim and mark external sources. If the documents leave part of the question open, say so. Context for this agent: The question and the documents that passed grading. Not the rejected documents or the grades. Procedure: 1. Confirm the index of your documents exists, and name the second source the system may use, such as web search on approved sites. 2. Search the index with the question and keep the top five documents. 3. Launch one grader per document in parallel, each in a fresh context. 4. If at least two documents are graded relevant, go to the writer step. 5. Otherwise launch the rewriter, run its query on the second source, and grade the new results the same way. Do this at most once. 6. Launch the writer with the question and every document graded relevant from either source. Stop rule: Stop after at most one fallback. If evidence is still thin after it, the writer answers from what passed and states what is missing. Exit: - `answer`, when: Enough relevant documents were found, or the fallback has run once. Deliver: The answer, the documents that passed grading with their sources, whether the fallback was used, and a note when evidence remained thin. # Final deliverable The answer with cited sources, each marked as internal or external, and one line stating whether the fallback ran and why.
The same thing in code
import asyncio
import operator
from typing import Annotated, TypedDict
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel
llm = init_chat_model("anthropic:claude-sonnet-5-5")
MIN_RELEVANT, SECOND_SOURCE = 2, "web search on approved sites"
GRADER = ("You receive one question and one document. Say relevant only if the document contains information "
"that helps answer this specific question, and quote the sentence that does. Do not answer the question.")
REWRITER = ("The documents found did not answer the question. Write one search query for the second source, "
"using the terms that source is likely to use. Keep the meaning of the question exactly.")
WRITER = ("Answer the question using only the documents provided. Cite the source of each claim and mark "
"external sources. If internal and external documents disagree, internal wins. Say what stays open.")
class Grade(BaseModel):
relevant: bool
quote: str
class Doc(TypedDict):
source: str
text: str
origin: str # internal or external
class Input(TypedDict):
task: str
class State(Input):
docs: list[Doc]
relevant: Annotated[list[Doc], operator.add]
query: str
fallback_used: bool
answer: str
def search_index(query: str, k: int = 5) -> list[Doc]:
"""Stub: search your own document index."""
return [Doc(source="policy.pdf", text=f"text about {query}", origin="internal")][:k]
def search_second_source(query: str, k: int = 5) -> list[Doc]:
"""Stub: plug in web search restricted to approved sites, or another collection."""
return [Doc(source="example-site/page", text=f"text about {query}", origin="external")][:k]
def retrieve(state: State) -> dict:
return {"docs": search_index(state["task"]), "fallback_used": False}
async def grade(state: State) -> dict:
# one call per document, in parallel; each grader sees the question and a single document
async def one(doc: Doc) -> bool:
out = await llm.with_structured_output(Grade).ainvoke(
[("system", GRADER), ("user", f"Question: {state['task']}\nDocument: {doc['text']}")])
return Grade.model_validate(out).relevant
verdicts = await asyncio.gather(*(one(d) for d in state["docs"]))
return {"relevant": [d for d, ok in zip(state["docs"], verdicts) if ok]}
def evidence_good(state: State) -> str:
# the gate: enough relevant documents, or the single fallback already used
return "writer" if len(state["relevant"]) >= MIN_RELEVANT or state["fallback_used"] else "rewriter"
async def rewriter(state: State) -> dict:
# sees the question and the name of the second source, not the rejected documents
msg = await llm.ainvoke([("system", REWRITER), ("user", f"Question: {state['task']}\nSecond source: {SECOND_SOURCE}")])
return {"query": msg.text.strip()}
def fallback(state: State) -> dict:
return {"docs": search_second_source(state["query"]), "fallback_used": True}
async def writer(state: State) -> dict:
docs = "\n".join(f"[{d['origin']}] {d['source']}: {d['text']}" for d in state["relevant"]) or "(none)"
msg = await llm.ainvoke([("system", WRITER), ("user", f"Question: {state['task']}\nDocuments:\n{docs}")])
return {"answer": msg.text}
builder = StateGraph(State, input_schema=Input)
builder.add_node("retrieve", retrieve)
builder.add_node("grade", grade)
builder.add_node("rewriter", rewriter)
builder.add_node("fallback", fallback)
builder.add_node("writer", writer)
builder.add_edge(START, "retrieve")
builder.add_edge("retrieve", "grade")
builder.add_conditional_edges("grade", evidence_good, ["writer", "rewriter"])
builder.add_edge("rewriter", "fallback")
builder.add_edge("fallback", "grade")
builder.add_edge("writer", END)
graph = builder.compile()
async def run(task: str):
s = await graph.ainvoke({"task": task})
why = "index gave too few relevant documents" if s["fallback_used"] else "index was enough"
return {"answer": s["answer"], "sources": [(d["source"], d["origin"]) for d in s["relevant"]],
"fallback": f"{'ran' if s['fallback_used'] else 'not run'}: {why}"}
Replace both search stubs with your own index and an approved second source, and keep the single-fallback limit when you do.
from typing import Any
from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
MIN_RELEVANT, SECOND_SOURCE = 2, "web search on approved sites"
AGENTS = {
"grader": AgentDefinition(
description="Grades one document against the question: relevant or irrelevant.",
prompt=("You receive one question and one document. Say relevant only if the document contains information "
"that helps answer this specific question, and quote the sentence that does. Do not answer the question."),
tools=[]),
"rewriter": AgentDefinition(
description="Writes one search query for the second source.",
prompt=("The documents found did not answer the question. Write one search query for the second source, "
"using the terms that source is likely to use. Keep the meaning of the question exactly."),
tools=[]),
"writer": AgentDefinition(
description="Answers from the documents that passed grading, with citations.",
prompt=("Answer the question using only the documents provided. Cite the source of each claim and mark "
"external sources. If internal and external documents disagree, internal wins. Say what stays open."),
tools=[]),
}
GRADES = {"type": "object", "required": ["relevant"], "properties": {"relevant": {"type": "array", "items": {"type": "boolean"}}}}
def search_index(query: str, k: int = 5) -> list[dict[str, str]]:
"""Stub: search your own document index."""
return [{"source": "policy.pdf", "text": f"text about {query}", "origin": "internal"}][:k]
def search_second_source(query: str, k: int = 5) -> list[dict[str, str]]:
"""Stub: plug in web search restricted to approved sites, or another collection."""
return [{"source": "example-site/page", "text": f"text about {query}", "origin": "external"}][:k]
async def ask(prompt: str, schema: dict[str, Any] | None = None) -> Any:
fmt = {"type": "json_schema", "schema": schema} if schema else None
options = ClaudeAgentOptions(agents=AGENTS, tools=["Agent"], allowed_tools=["Agent"], output_format=fmt)
async for message in query(prompt=prompt, options=options):
if isinstance(message, ResultMessage):
return message.structured_output if schema else message.result
raise RuntimeError("no result")
async def grade(task: str, docs: list[dict[str, str]]) -> list[dict[str, str]]:
# one grader sub-agent per document, in parallel, each seeing one document only
listing = "\n".join(f"{i}. {d['text']}" for i, d in enumerate(docs))
out = await ask(f"Question: {task}\nLaunch one grader sub-agent per document below, in parallel. Give each "
f"only the question and its own document.\n{listing}\nReturn the verdicts in document order.", GRADES)
return [d for d, ok in zip(docs, out["relevant"]) if ok]
async def run(task: str):
relevant = await grade(task, search_index(task))
fallback = len(relevant) < MIN_RELEVANT
if fallback: # the gate: at most one fallback, never a second correction
q = await ask(f"Use the rewriter sub-agent. Give it only:\nQuestion: {task}\nSecond source: {SECOND_SOURCE}\n"
"Return only its query.")
relevant += await grade(task, search_second_source(str(q).strip()))
docs = "\n".join(f"[{d['origin']}] {d['source']}: {d['text']}" for d in relevant) or "(none)"
answer = await ask(f"Use the writer sub-agent. Give it only:\nQuestion: {task}\nDocuments:\n{docs}\n"
"Return its answer verbatim.")
why = "index gave too few relevant documents" if fallback else "index was enough"
return {"answer": answer, "sources": [(d["source"], d["origin"]) for d in relevant],
"fallback": f"{'ran' if fallback else 'not run'}: {why}"}
Replace both search stubs with your own index and an approved second source, and keep the single-fallback limit when you do.
import asyncio
from agents import Agent, Runner
from pydantic import BaseModel
MIN_RELEVANT, SECOND_SOURCE = 2, "web search on approved sites"
class Grade(BaseModel):
relevant: bool
quote: str
grader = Agent(
name="Grader",
instructions=("You receive one question and one document. Say relevant only if the document contains information "
"that helps answer this specific question, and quote the sentence that does. Do not answer the question."),
output_type=Grade)
rewriter = Agent(
name="Query rewriter",
instructions=("The documents found did not answer the question. Write one search query for the second source, "
"using the terms that source is likely to use. Keep the meaning of the question exactly. Reply with the query only."))
writer = Agent(
name="Writer",
instructions=("Answer the question using only the documents provided. Cite the source of each claim and mark "
"external sources. If internal and external documents disagree, internal wins. Say what stays open."))
def search_index(query: str, k: int = 5) -> list[dict[str, str]]:
"""Stub: search your own document index."""
return [{"source": "policy.pdf", "text": f"text about {query}", "origin": "internal"}][:k]
def search_second_source(query: str, k: int = 5) -> list[dict[str, str]]:
"""Stub: plug in web search restricted to approved sites, or another collection."""
return [{"source": "example-site/page", "text": f"text about {query}", "origin": "external"}][:k]
async def grade(task: str, docs: list[dict[str, str]]) -> list[dict[str, str]]:
# one run per document, in parallel; a grader never sees the other documents
runs = await asyncio.gather(*(Runner.run(grader, f"Question: {task}\nDocument: {d['text']}") for d in docs))
return [d for d, r in zip(docs, runs) if r.final_output_as(Grade).relevant]
async def run(task: str):
relevant = await grade(task, search_index(task))
fallback = len(relevant) < MIN_RELEVANT
if fallback: # the gate: at most one fallback, never a second correction
q = await Runner.run(rewriter, f"Question: {task}\nSecond source: {SECOND_SOURCE}")
relevant += await grade(task, search_second_source(str(q.final_output).strip()))
docs = "\n".join(f"[{d['origin']}] {d['source']}: {d['text']}" for d in relevant) or "(none)"
out = await Runner.run(writer, f"Question: {task}\nDocuments:\n{docs}")
why = "index gave too few relevant documents" if fallback else "index was enough"
return {"answer": str(out.final_output), "sources": [(d["source"], d["origin"]) for d in relevant],
"fallback": f"{'ran' if fallback else 'not run'}: {why}"}
Replace both search stubs with your own index and an approved second source, and keep the single-fallback limit when you do.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
const model = anthropic('claude-sonnet-5-5');
const MIN_RELEVANT = 2, SECOND_SOURCE = 'web search on approved sites';
const GRADER =
'You receive one question and one document. Say relevant only if the document contains information that ' +
'helps answer this specific question, and quote the sentence that does. Do not answer the question.';
const REWRITER =
'The documents found did not answer the question. Write one search query for the second source, using the ' +
'terms that source is likely to use. Keep the meaning of the question exactly. Reply with the query only.';
const WRITER =
'Answer the question using only the documents provided. Cite the source of each claim and mark external ' +
'sources. If internal and external documents disagree, internal wins. Say what stays open.';
type Doc = { source: string; text: string; origin: 'internal' | 'external' };
// Stub: search your own document index.
async function searchIndex(query: string, k = 5): Promise<Doc[]> {
return [{ source: 'policy.pdf', text: `text about ${query}`, origin: 'internal' as const }].slice(0, k);
}
// Stub: plug in web search restricted to approved sites, or another collection.
async function searchSecondSource(query: string, k = 5): Promise<Doc[]> {
return [{ source: 'example-site/page', text: `text about ${query}`, origin: 'external' as const }].slice(0, k);
}
async function grade(task: string, docs: Doc[]): Promise<Doc[]> {
// one call per document, in parallel; a grader never sees the other documents
const verdicts = await Promise.all(docs.map(async (d) => (await generateText({
model, instructions: GRADER,
prompt: `Question: ${task}\nDocument: ${d.text}`,
output: Output.object({ schema: z.object({ relevant: z.boolean(), quote: z.string() }) }),
})).output.relevant));
return docs.filter((_, i) => verdicts[i]);
}
export async function run(task: string) {
const relevant = await grade(task, await searchIndex(task));
const fallback = relevant.length < MIN_RELEVANT;
if (fallback) { // the gate: at most one fallback, never a second correction
const { text: q } = await generateText({
model, instructions: REWRITER, // sees the question and the source name, not the rejected documents
prompt: `Question: ${task}\nSecond source: ${SECOND_SOURCE}`,
});
relevant.push(...(await grade(task, await searchSecondSource(q.trim()))));
}
const docs = relevant.map((d) => `[${d.origin}] ${d.source}: ${d.text}`).join('\n') || '(none)';
const { text } = await generateText({ model, instructions: WRITER, prompt: `Question: ${task}\nDocuments:\n${docs}` });
return {
answer: text,
sources: relevant.map((d) => ({ source: d.source, origin: d.origin })),
fallback: fallback ? 'ran: index gave too few relevant documents' : 'not run: index was enough',
};
}
Replace both search stubs with your own index and an approved second source, and keep the single-fallback limit when you do.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- A grader that sees all documents at once grades them against each other, not against the question. Grade one document per call.
- Rewriting the query until something passes turns a quality check into a search for confirmation. Allow one correction only.
- External sources may contradict your own documents. Tell the writer that your documents win when the two disagree.
Chain it
Its result can feed
- Reflection conditional
Condition: Hand over only the answer, without the graded evidence, and only when that answer is a written explanation rather than a short value such as a number, name or label.
- Chain of Verification conditional
Condition: Hand over only the answer part of the payload; the graded evidence list is not accepted by the entry.
- Debate conditional
Condition: Hand over only the answer part of the payload; the graded evidence list is not accepted by the entry.
- Constitutional Critique conditional
Condition: Hand over only the answer part of the payload; the graded evidence list is not accepted by the entry.
- Textual Backpropagation conditional
Condition: The link fits only the answer part of A's payload, since B's entry accepts an answer as material to improve but not the graded evidence bundle.