1Why pay for a research process when most questions need one lookup or none?
Questions differ in difficulty, but most systems treat them all the same way: either too lightly for the hard ones or too heavily for the easy ones. Adaptive RAG classifies each question first and routes it to no retrieval, one retrieval, or several rounds of retrieval.
You may need it if
- Your assistant answers "what does SSO stand for?" with the same slow search as a question comparing three contracts.
- Your model bill grows with traffic, but most questions are short and simple.
- Hard questions get shallow answers because the system only ever runs one search.
2When is a classifier in front just an extra step?
Use it when
- Your traffic mixes trivial questions with questions that need several documents combined.
- Cost or response time matters at your volume.
- You can give examples of easy, medium and hard questions from your own logs.
Avoid it when
- All questions are of the same kind, such as order status lookups. Pick the one route that fits and drop the classifier.
- Most questions are hard and need several rounds of searching anyway.
Use Agentic RAG instead - Questions are about the whole collection, such as the main themes across all customer interviews.
Use GraphRAG instead - The risk is not effort but accuracy, and every claim must be checked against a source.
Use Self-RAG instead
What it costs
- Model calls
- 1 to classify + 1 direct, 1 to 2 single-step, or up to 6 multi-step
- Latency
- medium
- Complexity
- medium
3How does a question find its route?
The classifier sorts the question into one of three routes. Simple questions are answered directly. Single-fact questions get one search and a grounded answer. Complex questions go to a researcher that searches in several steps, with a hop limit, before the same writer answers.
- Step 1 of 5
Sort before spending
A small, cheap classifier reads the question and picks a route. It is given examples of each kind from your own traffic. When in doubt, it picks the more thorough route.
- Step 2 of 5
Simple questions, no search
"What does SSO stand for?" needs no documents. The direct route answers in one call and is the fastest path through the system.
- Step 3 of 5
One fact, one search
"What is the refund window?" needs one passage. One search, then the writer answers from what it returned. This is classic retrieval, used only where it fits.
- Step 4 of 5
Several facts, several hops
"Which of our plans include SSO and cost less than the competitor's?" needs facts that build on each other. The researcher searches step by step until each step is answered or the hop limit is hit.
- Step 5 of 5
One answer format, whatever the route
Both retrieval routes end with the same writer, and every answer states which route it took. That label is what lets you check the classifier later.
What each part does
- Question
- The user's question in plain language.
- How complex?
- Sorts the question into simple, single-step or multi-step, rounding up when unsure.
- Direct answer
- Answers simple questions without searching, such as definitions or rewording.
- One search
- Runs a single search on your documents and returns the top passages.
- Researcher
- Breaks a complex question into steps and runs one search per step, using earlier findings.
- Enough found?
- Checks whether every step is answered, and counts the hops used.
- Writer
- Writes the answer from the passages found, citing them.
- Answer
- The answer, the route that was taken and the sources when any were used.
- How complex? → Direct answer: simple
- How complex? → One search: single-step
- How complex? → Researcher: multi-step
- Researcher → Enough found?: findings so far
- Enough found? → Researcher: steps open, hops left
- Enough found? → Writer: all steps done, or hops spent
- One search → Writer: passages
Where it connects
A question in plain language. Optionally, a forced route for testing or for questions known to be hard.
When: The chosen route has produced an answer, or the multi-step route has spent its hops.
The answer, the route taken (simple, single-step or multi-step), and the sources used when the route involved a search.
4What do I ask my assistant to set up?
The agents to launch
Classifier
Sort the question into one route. Simple: answerable without our documents. Single-step: one fact from our documents. Multi-step: several facts, where one depends on another. If unsure between two routes, choose the more thorough one. Reply with the route and one line of reason.
Sees: The question and a few labelled example questions for each route. Not the documents.
Direct answerer
Answer the question briefly from general knowledge. If you find it needs facts specific to our company or product, say so instead of guessing.
Sees: The question only. No documents.
Researcher
Break the question into the steps it depends on. For each open step, write one search query, using what earlier steps found. After each search, mark which steps are answered. Do not write the final answer.
Sees: The question, the search results so far and the list of open steps. Not the classifier's reasoning.
Writer
Answer the question from the passages provided and cite each one you use. If the passages leave part of the question open, say which part.
Sees: The question and the passages from the search or the researcher. Not the researcher's notes.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Adaptive RAG" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Adaptive RAG Judge how hard the question is, then spend only what it needs. Input: the task above. It must provide: A question in plain language. Optionally, a forced route for testing or for questions known to be hard. If something essential is missing, ask for it before starting. Agents to launch: - Classifier: Sort the question into one route. Simple: answerable without our documents. Single-step: one fact from our documents. Multi-step: several facts, where one depends on another. If unsure between two routes, choose the more thorough one. Reply with the route and one line of reason. Context for this agent: The question and a few labelled example questions for each route. Not the documents. - Direct answerer: Answer the question briefly from general knowledge. If you find it needs facts specific to our company or product, say so instead of guessing. Context for this agent: The question only. No documents. - Researcher: Break the question into the steps it depends on. For each open step, write one search query, using what earlier steps found. After each search, mark which steps are answered. Do not write the final answer. Context for this agent: The question, the search results so far and the list of open steps. Not the classifier's reasoning. - Writer: Answer the question from the passages provided and cite each one you use. If the passages leave part of the question open, say which part. Context for this agent: The question and the passages from the search or the researcher. Not the researcher's notes. Procedure: 1. Collect ten to twenty real questions from your logs and label each as simple, single-step or multi-step. Give them to the classifier as examples. 2. Launch the classifier on the question. 3. If simple, launch the direct answerer. If it reports that company facts are needed, reroute to single-step. 4. If single-step, run one search on your documents and launch the writer with the top five passages. 5. If multi-step, launch the researcher and run each query it writes. Repeat while steps are open, up to 4 hops, then launch the writer with all useful passages. 6. Label the answer with the route taken. Stop rule: Every route runs once. The multi-step route stops when all steps are answered or after 4 hops. A direct answer may be rerouted to single-step once, never more. Exit: - `answer`, when: The chosen route has produced an answer, or the multi-step route has spent its hops. Deliver: The answer, the route taken (simple, single-step or multi-step), and the sources used when the route involved a search. # Final deliverable The answer, the route taken with the classifier's one-line reason, and the sources used when a search ran.
The same thing in code
import operator
from typing import Annotated, Literal, TypedDict
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel
llm = init_chat_model("anthropic:claude-sonnet-5-5")
MAX_HOPS = 4
EXAMPLES = ("simple: What does SSO stand for?\n"
"single-step: What is our refund window?\n"
"multi-step: Which plans include SSO and cost less than the competitor's?")
def search_docs(query: str, k: int = 5) -> list[str]:
return [] # plug in your document search (vector store, BM25, ...)
class Route(BaseModel):
route: Literal["simple", "single-step", "multi-step"]
reason: str
class Direct(BaseModel):
answer: str
needs_company_facts: bool
class Step(BaseModel):
open_steps: list[str]
query: str | None
class State(TypedDict):
question: str
route: str
reason: str
answer: str
passages: Annotated[list[str], operator.add]
open_steps: list[str]
hops: int
async def ask[T: BaseModel](schema: type[T], prompt: str) -> T:
out = await llm.with_structured_output(schema).ainvoke(prompt)
assert isinstance(out, schema)
return out
async def classify(state: State) -> dict:
# the classifier sees labelled examples, never the documents
r = await ask(Route, f"Sort the question into simple, single-step or multi-step. "
f"If unsure, choose the more thorough route.\nExamples:\n{EXAMPLES}\n\nQuestion: {state['question']}")
return {"route": r.route, "reason": r.reason}
async def direct(state: State) -> dict:
d = await ask(Direct, "Answer briefly from general knowledge. If it needs facts about our company "
f"or product, set needs_company_facts instead of guessing.\n\n{state['question']}")
if d.needs_company_facts: # reroute to single-step, once
return {"route": "single-step", "reason": state["reason"] + " (rerouted from simple)"}
return {"answer": d.answer}
async def search(state: State) -> dict:
return {"passages": search_docs(state["question"])}
async def researcher(state: State) -> dict:
step = await ask(Step, "Break the question into the steps it depends on. List the steps still open "
"and write one search query for the next one (none if all are answered). Do not answer.\n\n"
f"Question: {state['question']}\nOpen steps: {state['open_steps']}\nResults so far: {state['passages']}")
found = search_docs(step.query) if step.query else []
return {"open_steps": step.open_steps, "passages": found, "hops": state["hops"] + 1}
async def writer(state: State) -> dict:
msg = await llm.ainvoke("Answer from these passages and cite each one you use. Say which part stays open.\n\n"
f"Question: {state['question']}\nPassages: {state['passages']}")
return {"answer": msg.text}
def by_route(state: State) -> Literal["direct", "search", "researcher"]:
if state["route"] == "simple":
return "direct"
return "search" if state["route"] == "single-step" else "researcher"
def after_direct(state: State) -> str:
return "search" if state["route"] == "single-step" else END
def enough_found(state: State) -> Literal["researcher", "writer"]:
# the hop gate: steps open and hops left, else write
return "researcher" if state["open_steps"] and state["hops"] < MAX_HOPS else "writer"
g = StateGraph(State)
g.add_node("classify", classify)
g.add_node("direct", direct)
g.add_node("search", search)
g.add_node("researcher", researcher)
g.add_node("writer", writer)
g.add_edge(START, "classify")
g.add_conditional_edges("classify", by_route)
g.add_conditional_edges("direct", after_direct, ["search", END])
g.add_edge("search", "writer")
g.add_conditional_edges("researcher", enough_found)
g.add_edge("writer", END)
app = g.compile()
async def run(task: str):
state = await app.ainvoke({"question": task, "route": "", "reason": "", "answer": "",
"passages": [], "open_steps": [], "hops": 0})
return {"answer": state["answer"], "route": state["route"], "reason": state["reason"], "sources": state["passages"]}
Replace search_docs with your retriever and give the classifier ten to twenty labelled questions from your own logs.
from typing import Any
from claude_agent_sdk import (AgentDefinition, ClaudeAgentOptions, ResultMessage,
create_sdk_mcp_server, query, tool)
MAX_HOPS = 4
passages: list[str] = []
hops = [0]
def search_docs(q: str, k: int = 5) -> list[str]:
return [] # plug in your document search (vector store, BM25, ...)
@tool("search", "Search our documents and return the top passages", {"query": str})
async def search(args: dict[str, Any]) -> dict[str, Any]:
if hops[0] >= MAX_HOPS: # the hop gate: 4 searches, then the researcher must stop
return {"content": [{"type": "text", "text": "Hop limit reached. Stop searching."}], "is_error": True}
hops[0] += 1
found = search_docs(args["query"])
passages.extend(found)
return {"content": [{"type": "text", "text": "\n\n".join(found) or "No results."}]}
DOCS = create_sdk_mcp_server("docs", tools=[search])
AGENTS = {
"classifier": AgentDefinition(
description="Sorts a question into simple, single-step or multi-step.",
prompt="Sort the question into one route. Simple: no documents needed. Single-step: one fact "
"from our documents. Multi-step: several dependent facts. If unsure, choose the more "
"thorough route. Give the route and one line of reason.", tools=[]),
"direct": AgentDefinition(
description="Answers simple questions without documents.",
prompt="Answer briefly from general knowledge. If it needs facts about our company or product, "
"say so instead of guessing.", tools=[]),
"researcher": AgentDefinition(
description="Searches step by step for multi-step questions.",
prompt="Break the question into the steps it depends on. For each open step, run one search, "
"using what earlier steps found. Stop when every step is answered or search refuses. "
"Do not write the final answer.", tools=["mcp__docs__search"]),
"writer": AgentDefinition(
description="Writes the cited answer from passages.",
prompt="Answer from the passages provided and cite each one you use. Say which part stays open.",
tools=[]),
}
async def delegate(agent: str, payload: str, schema: dict[str, Any] | None = None) -> Any:
options = ClaudeAgentOptions(
agents=AGENTS, mcp_servers={"docs": DOCS}, allowed_tools=["Agent", "mcp__docs__search"],
output_format={"type": "json_schema", "schema": schema} if schema else None)
prompt = f"Use the {agent} agent on the input below and return its reply unchanged.\n\n{payload}"
async for msg in query(prompt=prompt, options=options):
if isinstance(msg, ResultMessage):
return msg.structured_output if schema else msg.result
ROUTE = {"type": "object", "required": ["route", "reason"], "properties": {
"route": {"enum": ["simple", "single-step", "multi-step"]}, "reason": {"type": "string"}}}
DIRECT = {"type": "object", "required": ["answer", "needs_company_facts"], "properties": {
"answer": {"type": "string"}, "needs_company_facts": {"type": "boolean"}}}
async def run(task: str):
passages.clear()
hops[0] = 0
r = await delegate("classifier", task, ROUTE)
route, reason = r["route"], r["reason"]
if route == "simple":
d = await delegate("direct", task, DIRECT)
if not d["needs_company_facts"]:
return {"answer": d["answer"], "route": route, "reason": reason, "sources": []}
route = "single-step" # rerouted once, never more
if route == "single-step":
passages.extend(search_docs(task))
else:
await delegate("researcher", task) # its notes stay inside the sub-agent
# the writer sees the question and passages only
answer = await delegate("writer", f"Question: {task}\n\nPassages:\n" + "\n\n".join(passages))
return {"answer": answer, "route": route, "reason": reason, "sources": list(passages)}
Replace search_docs with your retriever; the module-level passage list and hop counter assume one run at a time.
from typing import Literal
from agents import Agent, Runner
from pydantic import BaseModel
MAX_HOPS = 4
def search_docs(query: str, k: int = 5) -> list[str]:
return [] # plug in your document search (vector store, BM25, ...)
class Route(BaseModel):
route: Literal["simple", "single-step", "multi-step"]
reason: str
class Direct(BaseModel):
answer: str
needs_company_facts: bool
class Step(BaseModel):
open_steps: list[str]
query: str | None
classifier = Agent(
name="Classifier", output_type=Route,
instructions="Sort the question into simple (no documents needed), single-step (one fact from our "
"documents) or multi-step (several dependent facts). If unsure, choose the more thorough "
"route. Give one line of reason. Examples: 'What does SSO stand for?' is simple.")
direct = Agent(
name="Direct answerer", output_type=Direct,
instructions="Answer briefly from general knowledge. If it needs facts about our company or product, "
"set needs_company_facts instead of guessing.")
researcher = Agent(
name="Researcher", output_type=Step,
instructions="Break the question into the steps it depends on. List the steps still open and write one "
"search query for the next one, using earlier results. Leave query empty when all are "
"answered. Do not write the final answer.")
writer = Agent(
name="Writer",
instructions="Answer from the passages provided and cite each one you use. Say which part stays open.")
async def run(task: str):
r = (await Runner.run(classifier, task)).final_output_as(Route)
route = r.route
passages: list[str] = []
if route == "simple":
d = (await Runner.run(direct, task)).final_output_as(Direct)
if not d.needs_company_facts:
return {"answer": d.answer, "route": route, "reason": r.reason, "sources": []}
route = "single-step" # rerouted once, never more
if route == "single-step":
passages = search_docs(task)
else:
open_steps: list[str] = []
for _ in range(MAX_HOPS): # the hop gate: at most 4 searches
# the researcher sees the question, results and open steps, not the classifier's reasoning
step = (await Runner.run(researcher, f"Question: {task}\nOpen steps: {open_steps}\n"
f"Results so far: {passages}")).final_output_as(Step)
open_steps = step.open_steps
if not step.query or not open_steps:
break
passages += search_docs(step.query)
# the writer sees passages, not the researcher's notes
out = await Runner.run(writer, f"Question: {task}\n\nPassages:\n" + "\n\n".join(passages))
return {"answer": out.final_output, "route": route, "reason": r.reason, "sources": passages}
Replace search_docs with your retriever and put real labelled examples from your logs in the classifier's instructions.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
const model = anthropic('claude-sonnet-5-5');
const MAX_HOPS = 4;
async function searchDocs(query: string, k = 5): Promise<string[]> {
return []; // plug in your document search (vector store, BM25, ...)
}
const Route = z.object({ route: z.enum(['simple', 'single-step', 'multi-step']), reason: z.string() });
const Direct = z.object({ answer: z.string(), needsCompanyFacts: z.boolean() });
const Step = z.object({ openSteps: z.array(z.string()), query: z.string().nullable() });
export async function run(task: string) {
// the classifier sees the question and labelled examples, never the documents
const { output: r } = await generateText({
model,
output: Output.object({ schema: Route }),
instructions:
'Sort the question into simple (no documents needed), single-step (one fact from our documents) ' +
'or multi-step (several dependent facts). If unsure, choose the more thorough route. ' +
"Give one line of reason. Example: 'What does SSO stand for?' is simple.",
prompt: task,
});
let route = r.route;
let passages: string[] = [];
if (route === 'simple') {
const { output: d } = await generateText({
model,
output: Output.object({ schema: Direct }),
instructions:
'Answer briefly from general knowledge. If it needs facts about our company or product, ' +
'set needsCompanyFacts instead of guessing.',
prompt: task,
});
if (!d.needsCompanyFacts) return { answer: d.answer, route, reason: r.reason, sources: [] };
route = 'single-step'; // rerouted once, never more
}
if (route === 'single-step') {
passages = await searchDocs(task);
} else {
let openSteps: string[] = [];
for (let hop = 0; hop < MAX_HOPS; hop++) { // the hop gate: at most 4 searches
const { output: step } = await generateText({
model,
output: Output.object({ schema: Step }),
instructions:
'Break the question into the steps it depends on. List the steps still open and write one ' +
'search query for the next one, using earlier results. Null query when all are answered. ' +
'Do not write the final answer.',
prompt: `Question: ${task}\nOpen steps: ${JSON.stringify(openSteps)}\nResults so far: ${passages.join('\n')}`,
});
openSteps = step.openSteps;
if (!step.query || openSteps.length === 0) break;
passages.push(...(await searchDocs(step.query)));
}
}
// the writer sees the passages, not the researcher's notes
const { text } = await generateText({
model,
instructions: 'Answer from the passages provided and cite each one you use. Say which part stays open.',
prompt: `Question: ${task}\n\nPassages:\n${passages.join('\n\n')}`,
});
return { answer: text, route, reason: r.reason, sources: passages };
}
Replace searchDocs with your retriever and put real labelled examples from your logs in the classifier's instructions.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- A classifier with no examples from your traffic guesses by question length. Short questions can be the hardest ones.
- Without a route label on each answer you cannot audit the classifier. Always record it.
- If the direct route is allowed to answer company-specific questions, it invents policies. Make it hand those back.
Chain it
Its result can feed
- Reflection conditional
Condition: Use this link only when A's answer is a substantive prose response, not a bare fact such as a number, name or label.
- Chain of Verification direct
Treat A's answer as the baseline draft. Turn its factual claims into verification questions, answer each one independently, and keep, correct or remove each claim.
- Debate direct
Treat A's answer as the existing answer to challenge on the original question. Debaters defend, correct or abandon it, and the judge settles what remains in dispute.
- Constitutional Critique direct
Treat A's answer as the text to check. Test it against every rule of the stored constitution, and fix only the failed rules while keeping its sourced content.
- Textual Backpropagation conditional
Condition: Build the link only when A's answer is an extended written response, not a short factual reply such as a number, name or label.