1Why pay for the strongest model on questions a small one gets right?
Models are placed on a ladder, cheapest first. Every question starts on the first rung. A scorer rates the answer that came back. If the score clears that rung's threshold, the answer is returned. If not, the same question goes to the next model. The last rung's answer is returned as it is.
You may need it if
- Most questions are easy, yet all of them go to your most expensive model.
- You cannot tell from the wording of a question whether a small model will get it right.
- You have past questions with known answers, enough to learn when an answer can be relied on.
2When does trying the cheap model first cost more than it saves?
Use it when
- A large share of requests can be answered well by the first model on the ladder.
- The price gap between the first and the last model is wide.
- You have labelled examples, close to real traffic, to train or calibrate the scorer and set the thresholds.
- A slower reply on the harder questions is acceptable.
Avoid it when
- The kind of request, readable from its wording, decides which machinery fits. Classify once and dispatch.
Use Meta-Controller instead - The real question is whether to answer at all, given what a wrong answer would cost.
Use Metacognitive Triage instead - You want several models to answer and their answers combined, not the first acceptable one returned.
Use Ensemble instead - Most requests end on the last rung anyway. Each of them has paid for the failed attempts and waited for them.
- Nothing reliable can score an answer for this task, or you have no labelled examples to set the thresholds.
What it costs
- Model calls
- 1 answer + 1 score per rung tried; the last rung is not scored
- Latency
- medium
- Complexity
- medium
3How does a question climb the ladder?
A ladder with a check on each rung. The first model answers and a scorer rates the answer. A threshold decides between returning it and sending the same question to the next model. Nothing classifies the question beforehand. The last model's answer leaves without a check.
- Step 1 of 6
Every question starts on the first rung
A customer asks whether the delivery address can still be changed after the parcel has shipped. Nothing sorts the question first. The cheapest model answers it.
- Step 2 of 6
A scorer rates the answer
A small scoring model reads the question and the answer and returns 0.93. It does not ask the answering model how sure it is.
- Step 3 of 6
The threshold lets it through
The first rung's threshold is 0.9. The answer clears it and is returned. The stronger models were never called. Most questions should end here.
- Step 4 of 6
A weak answer sends the question up
Another customer asks about customs fees on a return from abroad. The first answer scores 0.41. It is dropped, and the same question goes to the next model.
- Step 5 of 6
The next rung is checked the same way
On a ladder of three, the second model's answer is scored against its own threshold, here 0.7. It passes or the question climbs again.
- Step 6 of 6
The last rung answers without a check
When the question reaches the last model, its answer is returned as it is. There is nowhere further to send it. The reply says which rung produced it.
What each part does
- Question
- The question or small task, unchanged from rung to rung.
- Cheapest model
- The first rung. Answers every question that comes in.
- Answer scorer
- Rates from 0 to 1 how reliable an answer is, from the question and the answer. A small model trained for this.
- Good enough?
- Plain code. Compares the score with the threshold of the rung that produced the answer.
- Next model
- The following rungs, in a fixed order. Each gets the same question, without the rejected answer.
- Answer
- The first accepted answer, or the last rung's answer, with the rung that produced it.
- Cheapest model → Answer scorer: answer
- Answer scorer → Good enough?: reliability score
- Good enough? → Answer: score clears the threshold
- Good enough? → Next model: below the threshold
- Next model → Answer scorer: more rungs remain
- Next model → Answer: last rung
Where it connects
A question or small task that one model call can answer, of a kind the scorer was calibrated on. The ladder and its thresholds are already set.
When: An answer cleared its rung's threshold, or the last rung answered.
The answer, the rung that produced it, and the scores of the answers that were rejected on the way.
4What do I ask my assistant to set up?
The agents to launch
First model
Answer the question directly and briefly. This role runs on the cheapest model of the ladder. It gets no hint that its answer will be scored or that stronger models exist.
Sees: The question only. Not the thresholds and not the rest of the ladder.
Next model, one per further rung, usually 1 or 2
Answer the question directly and briefly. This role runs on the next model of the ladder, each rung stronger and more expensive than the one before. Answer from the question alone.
Sees: The question only. Not the rejected answer and not its score, so that an earlier mistake does not carry over.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "LLM Cascade" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # LLM Cascade Try the cheapest model first and climb only when its answer fails. Input: the task above. It must provide: A question or small task that one model call can answer, of a kind the scorer was calibrated on. The ladder and its thresholds are already set. If something essential is missing, ask for it before starting. Agents to launch: - First model: Answer the question directly and briefly. This role runs on the cheapest model of the ladder. It gets no hint that its answer will be scored or that stronger models exist. Context for this agent: The question only. Not the thresholds and not the rest of the ladder. - Next model (one per further rung, usually 1 or 2): Answer the question directly and briefly. This role runs on the next model of the ladder, each rung stronger and more expensive than the one before. Answer from the question alone. Context for this agent: The question only. Not the rejected answer and not its score, so that an earlier mistake does not carry over. Procedure: 1. Order two or three models from cheapest to most capable. That order is the ladder, and it does not change per question. 2. Prepare the scorer: a small model that takes a question and an answer and returns a reliability score from 0 to 1. Train or calibrate it on past questions with known answers. 3. Set one threshold per rung except the last, on those same examples. For instance 0.9 on the first rung and 0.7 on the second. 4. Send the question to the first model, then score its answer. 5. If the score reaches the rung's threshold, return the answer. If not, drop it and send the unchanged question to the next model. 6. Return the last model's answer without scoring it. 7. Log the rung that answered each question. If most questions reach the last rung, remove the cascade. Stop rule: Stop at the first answer whose score reaches its rung's threshold, for example 0.9 then 0.7. On the last rung, at most the third, return the answer unscored. Never go back down. Exit: - `answer`, when: An answer cleared its rung's threshold, or the last rung answered. Deliver: The answer, the rung that produced it, and the scores of the answers that were rejected on the way. # Final deliverable The answer, the rung and model that produced it, and the score of each rejected answer.
The same thing in code
import operator
from typing import Annotated, TypedDict
from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel, Field
# the ladder, cheapest first, with the threshold of each rung; the last rung is never scored
LADDER = [("anthropic:claude-haiku-5-5", 0.9), ("anthropic:claude-sonnet-5-5", 0.0)]
models = [init_chat_model(name) for name, _ in LADDER]
# stand-in scorer: replace it with a small classifier trained on past questions with known answers
scorer_model = init_chat_model("anthropic:claude-haiku-5-5")
ANSWER = "Answer the question directly and briefly."
SCORER = "Rate from 0 to 1 how likely this answer to the question is correct and complete. Do not answer it yourself."
class Reliability(BaseModel):
score: float = Field(ge=0, le=1)
class State(TypedDict):
question: str
rung: int
answer: str
score: float
rejected: Annotated[list[str], operator.add]
async def answer(state: State) -> dict:
# every rung gets the question alone, not the rejected answer
reply = await models[state["rung"]].ainvoke([SystemMessage(ANSWER), HumanMessage(state["question"])])
return {"answer": reply.text}
def last_rung(state: State) -> str:
return END if state["rung"] == len(LADDER) - 1 else "score" # the last answer leaves unscored
async def score(state: State) -> dict:
prompt = f"Question: {state['question']}\n\nAnswer: {state['answer']}"
raw = await scorer_model.with_structured_output(Reliability).ainvoke([SystemMessage(SCORER), HumanMessage(prompt)])
return {"score": Reliability.model_validate(raw).score}
def good_enough(state: State) -> str: # the gate: plain code against the rung's threshold
return END if state["score"] >= LADDER[state["rung"]][1] else "climb"
def climb(state: State) -> dict:
return {"rung": state["rung"] + 1, "rejected": [f"rung {state['rung'] + 1}: score {state['score']:.2f}"]}
builder = StateGraph(State)
builder.add_node("answer", answer)
builder.add_node("score", score)
builder.add_node("climb", climb)
builder.add_edge(START, "answer")
builder.add_conditional_edges("answer", last_rung, ["score", END])
builder.add_conditional_edges("score", good_enough, ["climb", END])
builder.add_edge("climb", "answer")
graph = builder.compile()
async def run(question: str) -> dict:
s = await graph.ainvoke({"question": question, "rung": 0, "answer": "", "score": 0.0, "rejected": []})
return {"answer": s["answer"], "rung": s["rung"] + 1, "model": LADDER[s["rung"]][0], "rejected": s["rejected"]}
The scorer here is a model call used as a placeholder. Train a small classifier on your own labelled questions, then set each threshold on the same data.
from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
from pydantic import BaseModel, Field
# the ladder, cheapest first, with the threshold of each rung; the last rung is never scored
LADDER = [("haiku", 0.9), ("sonnet", 0.0)]
class Reply(BaseModel):
answer: str
class Reliability(BaseModel):
score: float = Field(ge=0, le=1)
def rung_agent(model: str) -> AgentDefinition:
return AgentDefinition(
description=f"Answers the question. Runs on {model}.",
prompt="Answer the question directly and briefly.",
tools=[],
model=model,
)
SCORER = AgentDefinition(
# stand-in scorer: replace it with a small classifier trained on past questions with known answers
description="Rates how reliable an answer is.",
prompt="Rate from 0 to 1 how likely this answer to the question is correct and complete. Do not answer it yourself.",
tools=[],
model="haiku",
)
async def delegate(name: str, agent: AgentDefinition, payload: str, schema: type[BaseModel]) -> dict:
# one query per call: the agent sees only the payload
options = ClaudeAgentOptions(
agents={name: agent}, allowed_tools=["Agent"], max_turns=4,
output_format={"type": "json_schema", "schema": schema.model_json_schema()},
)
async for message in query(prompt=f"Hand this input unchanged to the {name} agent and return its result:\n\n{payload}", options=options):
if isinstance(message, ResultMessage):
return schema.model_validate(message.structured_output).model_dump()
raise RuntimeError(f"{name} returned nothing")
async def run(question: str) -> dict:
rejected: list[str] = []
for rung, (model, threshold) in enumerate(LADDER, start=1):
# every rung gets the question alone, not the rejected answer
answer = (await delegate(f"rung-{rung}", rung_agent(model), question, Reply))["answer"]
if rung == len(LADDER): # the last answer leaves unscored
return {"answer": answer, "rung": rung, "model": model, "rejected": rejected}
rated = await delegate("scorer", SCORER, f"Question: {question}\n\nAnswer: {answer}", Reliability)
if rated["score"] >= threshold: # the gate: plain code against the rung's threshold
return {"answer": answer, "rung": rung, "model": model, "rejected": rejected}
rejected.append(f"rung {rung}: score {rated['score']:.2f}")
raise RuntimeError("the ladder is empty")
The scorer here is a model call used as a placeholder. Train a small classifier on your own labelled questions, then set each threshold on the same data.
from agents import Agent, Runner
from pydantic import BaseModel, Field
# the ladder, cheapest first, with the threshold of each rung; the last rung is never scored
LADDER = [("your-cheapest-model", 0.9), ("your-strongest-model", 0.0)] # set the ids for your provider
SCORER_MODEL = "your-cheapest-model"
class Reliability(BaseModel):
score: float = Field(ge=0, le=1)
rungs = [
Agent(name=f"Rung {i}", model=model, instructions="Answer the question directly and briefly.")
for i, (model, _) in enumerate(LADDER, start=1)
]
# stand-in scorer: replace it with a small classifier trained on past questions with known answers
scorer = Agent(
name="Answer scorer", model=SCORER_MODEL, output_type=Reliability,
instructions="Rate from 0 to 1 how likely this answer to the question is correct and complete. "
"Do not answer it yourself.")
async def run(question: str) -> dict:
rejected: list[str] = []
for rung, (agent, (model, threshold)) in enumerate(zip(rungs, LADDER), start=1):
# every rung gets the question alone, not the rejected answer
answer = (await Runner.run(agent, question)).final_output_as(str)
if rung == len(LADDER): # the last answer leaves unscored
return {"answer": answer, "rung": rung, "model": model, "rejected": rejected}
rated = await Runner.run(scorer, f"Question: {question}\n\nAnswer: {answer}")
score = rated.final_output_as(Reliability).score
if score >= threshold: # the gate: plain code against the rung's threshold
return {"answer": answer, "rung": rung, "model": model, "rejected": rejected}
rejected.append(f"rung {rung}: score {score:.2f}")
raise RuntimeError("the ladder is empty")
Set the model ids of your ladder. The scorer here is a model call used as a placeholder for a small classifier trained on your own labelled questions.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
// the ladder, cheapest first, with the threshold of each rung; the last rung is never scored
const LADDER = [
{ id: 'claude-haiku-5-5', threshold: 0.9 },
{ id: 'claude-sonnet-5-5', threshold: 0 },
];
// stand-in scorer: replace it with a small classifier trained on past questions with known answers
const scorerModel = anthropic('claude-haiku-5-5');
const reliabilitySchema = z.object({ score: z.number().min(0).max(1) });
async function score(question: string, answer: string): Promise<number> {
const { output } = await generateText({
model: scorerModel,
instructions: 'Rate from 0 to 1 how likely this answer to the question is correct and complete. Do not answer it yourself.',
prompt: `Question: ${question}\n\nAnswer: ${answer}`,
output: Output.object({ schema: reliabilitySchema }),
});
return output.score;
}
export async function run(question: string) {
const rejected: string[] = [];
for (let i = 0; i < LADDER.length; i++) {
const { id, threshold } = LADDER[i];
// every rung gets the question alone, not the rejected answer
const { text: answer } = await generateText({
model: anthropic(id),
instructions: 'Answer the question directly and briefly.',
prompt: question,
});
if (i === LADDER.length - 1) return { answer, rung: i + 1, model: id, rejected }; // the last answer leaves unscored
const s = await score(question, answer);
if (s >= threshold) return { answer, rung: i + 1, model: id, rejected }; // the gate: plain code against the threshold
rejected.push(`rung ${i + 1}: score ${s.toFixed(2)}`);
}
throw new Error('the ladder is empty');
}
The scorer here is a model call used as a placeholder. Train a small classifier on your own labelled questions, then set each threshold on the same data.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- Simply asking the answering model whether it is sure accepts almost everything. Use a separate scorer, or a self-check calibrated for the purpose.
- Thresholds set on examples unlike real traffic accept wrong answers or escalate everything. Calibrate on your own questions.
- Passing the rejected answer upward can anchor the next model on the same mistake. Send the question alone.
- Each escalation adds the price and the wait of the failed attempt. Watch the share of questions that climb.
- Training the scorer and tuning the thresholds is work done before the first question. It pays back only at volume.
Chain it
Its result can feed
- Reflection conditional
Condition: Use this link only when the cascade's answer is a piece of prose or other written work, not a short answer such as a number, a label or a choice.
- Chain of Verification conditional
Condition: The task calls for factual information rather than a purely creative or subjective response.
- Debate direct
Treat the cascade's answer as an existing answer to challenge on the original question, and let the debaters defend or overturn it.
- Constitutional Critique direct
Treat the cascade's answer as the draft to check rule by rule against the constitution, fixing only the rules it fails.
- Textual Backpropagation conditional
Condition: Use this link only when the cascade's answer is a piece of prose or other written work, not a short answer such as a number, a label or a choice.