1How do I stop one agent's blind spot from becoming my answer?
Every agent has habits that lead it wrong in the same way each time. An ensemble asks several agents that are deliberately different, by instructions, role or model, to answer the same task alone. An aggregator combines them by vote or weighting, so mistakes that are not shared cancel out.
You may need it if
- One agent makes judgement calls, such as lead scoring or contract risk, and its errors lean one way.
- Two prompts that both look reasonable give different answers to the same case.
- The answer is a choice or a number that can be compared across agents.
2When does a panel of agents not beat one agent?
Use it when
- The task has a short answer that can be voted on or averaged, such as a category, a yes or no, or an estimate.
- You can design three to five genuinely different ways to approach it.
- A biased answer costs more than three to five times the model calls.
Avoid it when
- A single well-tested prompt exists and the only risk is a random slip. Sampling it several times is cheaper and simpler.
Use Self-Consistency instead - All voters lack the same fact. They will agree confidently on the wrong answer. Fetch the evidence instead.
Use Agentic RAG instead - The answer is a long text, such as a proposal or a report. Long texts cannot be voted on.
Use Reflection instead - You need the disagreement argued out and resolved, not just counted.
Use Debate instead
What it costs
- Model calls
- voters + 1 aggregation (typically 4 to 6)
- Latency
- low
- Complexity
- medium
3How do the voters and the aggregator work together?
A fan-out and a fan-in. The same task goes to three voters with different instructions or models. Each answers alone and never sees the others. The aggregator groups equivalent answers, applies the weights and reports the winner with the level of agreement.
- Step 1 of 4
One task, three angles
The same task, for example "will this trial account convert?", goes to three voters. Each one is built to look at it differently: from the facts, from base rates, through another model.
- Step 2 of 4
Answers in isolation
Each voter answers alone, in the required form, with a short reason. No voter sees another's answer, because the first one heard would anchor the rest.
- Step 3 of 4
Combine
The aggregator groups answers that mean the same thing, then counts them, weighted by each voter's past accuracy if you track it. Ties are reported, not hidden.
- Step 4 of 4
The answer and the split
The panel answer leaves with the vote split. A 3 to 0 result and a 2 to 1 result deserve different levels of trust, and the reader can see which one this is.
What each part does
- Task
- The task or question, with the form the answer must take, such as a label, a number or yes or no.
- Bottom-up analyst
- Works only from the facts given, step by step, and ignores what usually happens.
- Base-rate voter
- Starts from how similar cases usually turn out, then adjusts for what is unusual here.
- Second-model voter
- Runs the plain task on a different model family, so its training habits differ.
- Aggregator
- Groups answers that mean the same thing, applies the voter weights and measures the agreement.
- Panel answer
- The winning answer, the vote split and any dissent worth reading.
- Bottom-up analyst → Aggregator: answer and reason
- Base-rate voter → Aggregator: answer and reason
- Second-model voter → Aggregator: answer and reason
- Aggregator → Panel answer: weighted vote
Where it connects
A task or question whose answer is short enough to compare, such as a label, a choice, a number or yes or no. If the form is not stated, the stage fixes one.
When: Every voter has answered and the aggregator has combined the votes.
The winning answer, the vote split with weights, each voter's one-line reason, and the strongest dissenting argument when the vote was not unanimous.
4What do I ask my assistant to set up?
The agents to launch
Bottom-up analyst
Answer the task using only the facts provided. Reason step by step from those facts and ignore what is typical for cases like this. Give the answer in the required form, then one line saying which fact decided it.
Sees: The task, its facts and the required answer form. Nothing from the other voters.
Base-rate voter
Start from how cases like this usually turn out. State that typical outcome first. Then adjust it for what is unusual in this case, and only as far as the facts justify. Give the answer in the required form and one line of reason.
Sees: The task, its facts and the required answer form. Nothing from the other voters.
Second-model voter
Answer the task plainly. Give the answer in the required form, then one line of reason. Run this agent on a different model family from the other two voters.
Sees: The task, its facts and the required answer form. Nothing from the other voters.
Aggregator
Group answers that mean the same thing. Count the votes, applying the weights if given, otherwise one vote each. Report the winner and the split. On a tie, say so and give the answer the safer choice would be. Quote the strongest dissenting reason.
Sees: The task, the required answer form, the weights, and each voter's answer with its one-line reason. Not their full reasoning.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Ensemble" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Ensemble Deliberately different agents answer alone, then a vote decides. Input: the task above. It must provide: A task or question whose answer is short enough to compare, such as a label, a choice, a number or yes or no. If the form is not stated, the stage fixes one. If something essential is missing, ask for it before starting. Agents to launch: - Bottom-up analyst: Answer the task using only the facts provided. Reason step by step from those facts and ignore what is typical for cases like this. Give the answer in the required form, then one line saying which fact decided it. Context for this agent: The task, its facts and the required answer form. Nothing from the other voters. - Base-rate voter: Start from how cases like this usually turn out. State that typical outcome first. Then adjust it for what is unusual in this case, and only as far as the facts justify. Give the answer in the required form and one line of reason. Context for this agent: The task, its facts and the required answer form. Nothing from the other voters. - Second-model voter: Answer the task plainly. Give the answer in the required form, then one line of reason. Run this agent on a different model family from the other two voters. Context for this agent: The task, its facts and the required answer form. Nothing from the other voters. - Aggregator: Group answers that mean the same thing. Count the votes, applying the weights if given, otherwise one vote each. Report the winner and the split. On a tie, say so and give the answer the safer choice would be. Quote the strongest dissenting reason. Context for this agent: The task, the required answer form, the weights, and each voter's answer with its one-line reason. Not their full reasoning. Procedure: 1. Fix the form of the answer, such as one label from a list, a number in a range, or yes or no. Without a fixed form, answers cannot be counted. 2. Launch the three voters at the same time, each in a fresh context with its own brief. 3. Check that each answer has the required form. Ask a voter once to restate an answer that does not. 4. Launch the aggregator on the answers and one-line reasons. 5. Record each voter's answer so its accuracy can be measured later and used as a weight. Stop rule: Stop after one round of voting. Do not rerun voters to break a tie; report the tie and the safer choice instead. Exit: - `answer`, when: Every voter has answered and the aggregator has combined the votes. Deliver: The winning answer, the vote split with weights, each voter's one-line reason, and the strongest dissenting argument when the vote was not unanimous. # Final deliverable The panel answer, the vote split, each voter's answer with its reason, and a one-line note on how much to trust the result given the agreement.
The same thing in code
import operator
from typing import Annotated, TypedDict
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel
llm = init_chat_model("anthropic:claude-sonnet-5-5")
second_llm = init_chat_model("anthropic:claude-haiku-5-5") # swap for a model from another family
LABELS = ["approve", "reject", "escalate"] # the fixed answer form
WEIGHTS = {"bottom-up": 1.0, "base-rate": 1.0, "second-model": 1.0} # replace with measured accuracy
LEDGER: list[dict] = [] # each voter's answer, kept to measure accuracy later
VOTERS = {
"bottom-up": (llm, "Use only the facts provided. Reason step by step and ignore what is typical. "
"Give the answer, then one line naming the fact that decided it."),
"base-rate": (llm, "Start from how cases like this usually turn out and state it. Adjust only as far "
"as the facts justify. Give the answer and one line of reason."),
"second-model": (second_llm, "Answer the task plainly. Give the answer and one line of reason."),
}
class Vote(BaseModel):
answer: str
reason: str
class Panel(BaseModel):
winner: str
split: str
tie: bool
dissent: str
class State(TypedDict):
task: str
votes: Annotated[list[tuple[str, Vote]], operator.add] # fan-in: each voter appends its own
panel: Panel | None
def voter(name: str):
model, brief = VOTERS[name]
async def node(state: State) -> dict:
# a fresh context: the task only, nothing from the other voters
prompt = f"{brief}\nAnswer with one of {LABELS}.\n\nTask: {state['task']}"
vote = Vote.model_validate(await model.with_structured_output(Vote).ainvoke(prompt))
if vote.answer not in LABELS: # ask once to restate
vote = Vote.model_validate(await model.with_structured_output(Vote).ainvoke(
f"{prompt}\n\nYour answer '{vote.answer}' is not one of {LABELS}. Restate it."))
return {"votes": [(name, vote)]}
return node
async def aggregator(state: State) -> dict:
# sees each answer and its one-line reason, not the voters' full reasoning
ballots = "\n".join(f"- {n} (weight {WEIGHTS[n]}): {v.answer}. {v.reason}" for n, v in state["votes"])
LEDGER.extend({"task": state["task"], "voter": n, "answer": v.answer} for n, v in state["votes"])
prompt = (
"Group answers that mean the same thing, count weighted votes, report the winner and the split. "
"On a tie, say so and give the safer choice. Quote the strongest dissenting reason.\n"
f"Task: {state['task']}\nAllowed answers: {LABELS}\n{ballots}"
)
return {"panel": Panel.model_validate(await llm.with_structured_output(Panel).ainvoke(prompt))}
graph = StateGraph(State)
for name in VOTERS:
graph.add_node(name, voter(name))
graph.add_edge(START, name) # the fan-out: all three run in parallel
graph.add_node("aggregator", aggregator)
graph.add_edge(list(VOTERS), "aggregator") # waits for all three; one round only
graph.add_edge("aggregator", END)
app = graph.compile()
async def run(task: str):
final = await app.ainvoke({"task": task, "votes": [], "panel": None})
return final["panel"], final["votes"]
Point the second voter at a model from another provider, and replace the equal weights with each voter's measured accuracy once the ledger has enough cases.
from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
from pydantic import BaseModel
LABELS = ["approve", "reject", "escalate"] # the fixed answer form
WEIGHTS = {"bottom-up": 1.0, "base-rate": 1.0, "second-model": 1.0} # replace with measured accuracy
FORM = f"Give exactly one of {LABELS}, then one line of reason."
class Ballot(BaseModel):
voter: str
answer: str
reason: str
class Panel(BaseModel):
winner: str
split: str
tie: bool
dissent: str
ballots: list[Ballot]
trust: str
AGENTS = {
"bottom-up": AgentDefinition(
description="Voter that reasons only from the given facts.",
prompt=f"Use only the facts provided. Reason step by step and ignore what is typical. {FORM}",
tools=[],
),
"base-rate": AgentDefinition(
description="Voter that starts from how similar cases usually turn out.",
prompt=f"State how cases like this usually turn out, then adjust only as far as the facts justify. {FORM}",
tools=[],
),
"second-model": AgentDefinition(
description="Voter running on a different model.",
prompt=f"Answer the task plainly. {FORM}",
model="haiku", # point this at a model from another family when you can
tools=[],
),
"aggregator": AgentDefinition(
description="Groups equivalent answers and counts the weighted vote.",
prompt="Group answers that mean the same thing, count weighted votes, report the winner and the split. "
"On a tie, say so and give the safer choice. Quote the strongest dissenting reason.",
tools=[],
),
}
ORCHESTRATION = f"""Run a one-round vote on the task below.
1. Launch the bottom-up, base-rate and second-model agents in parallel. Give each only the task,
never another voter's answer.
2. If an answer is not one of {LABELS}, ask that voter once to restate it.
3. Give the aggregator the task, the allowed answers, the weights {WEIGHTS} and each answer with
its one-line reason only.
4. Stop after this one round, even on a tie. Return the panel result."""
async def run(task: str):
options = ClaudeAgentOptions(
agents=AGENTS,
tools=["Agent"],
allowed_tools=["Agent"],
output_format={"type": "json_schema", "schema": Panel.model_json_schema()},
max_turns=12,
)
async for message in query(prompt=f"{ORCHESTRATION}\n\nTask: {task}", options=options):
if isinstance(message, ResultMessage):
return Panel.model_validate(message.structured_output)
raise RuntimeError("the panel returned nothing")
The prompt enforces isolation here; for a hard guarantee, launch each voter in its own query and pass only the answers to the aggregator.
import asyncio
from typing import Literal
from agents import Agent, Runner
from pydantic import BaseModel
Label = Literal["approve", "reject", "escalate"] # the fixed answer form: votes can be counted
WEIGHTS = {"Bottom-up analyst": 1.0, "Base-rate voter": 1.0, "Second-model voter": 1.0} # use measured accuracy
LEDGER: list[tuple[str, str, str]] = [] # (task, voter, answer), to measure accuracy later
class Vote(BaseModel):
answer: Label
reason: str
class Panel(BaseModel):
winner: Label
split: str
tie: bool
dissent: str
voters = [
Agent(name="Bottom-up analyst", output_type=Vote,
instructions="Use only the facts provided. Reason step by step and ignore what is typical. "
"Give the answer and one line naming the fact that decided it."),
Agent(name="Base-rate voter", output_type=Vote,
instructions="State how cases like this usually turn out, then adjust only as far as the facts "
"justify. Give the answer and one line of reason."),
# set model= to a model from another provider so its habits differ
Agent(name="Second-model voter", output_type=Vote,
instructions="Answer the task plainly. Give the answer and one line of reason."),
]
aggregator = Agent(
name="Aggregator",
output_type=Panel,
instructions="Count the weighted votes, report the winner and the split. On a tie, say so and give "
"the safer choice. Quote the strongest dissenting reason.",
)
async def run(task: str):
# the fan-out: three separate runs in parallel, none sees another's answer
results = await asyncio.gather(*(Runner.run(v, task) for v in voters))
votes = [(v.name, r.final_output_as(Vote)) for v, r in zip(voters, results)]
LEDGER.extend((task, name, vote.answer) for name, vote in votes)
ballots = "\n".join(f"- {n} (weight {WEIGHTS[n]}): {v.answer}. {v.reason}" for n, v in votes)
# one round only: no rerun to break a tie
panel = await Runner.run(aggregator, f"Task: {task}\n\nBallots:\n{ballots}")
return panel.final_output_as(Panel), votes
Give the second voter a model from another family, and replace the equal weights with each voter's measured accuracy from the ledger.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
const LABELS = ['approve', 'reject', 'escalate'] as const; // the fixed answer form
const voteSchema = z.object({ answer: z.enum(LABELS), reason: z.string() });
const ledger: { task: string; voter: string; answer: string }[] = []; // to measure accuracy later
const voters = [
{
name: 'Bottom-up analyst',
model: anthropic('claude-sonnet-5-5'),
weight: 1,
instructions: 'Use only the facts provided. Reason step by step and ignore what is typical. ' +
'Give the answer and one line naming the fact that decided it.',
},
{
name: 'Base-rate voter',
model: anthropic('claude-sonnet-5-5'),
weight: 1,
instructions: 'State how cases like this usually turn out, then adjust only as far as the facts justify. ' +
'Give the answer and one line of reason.',
},
{
name: 'Second-model voter',
model: anthropic('claude-haiku-5-5'), // swap for another provider's model so its habits differ
weight: 1,
instructions: 'Answer the task plainly. Give the answer and one line of reason.',
},
];
export async function run(task: string) {
// the fan-out: three isolated calls in parallel, none sees another's answer
const votes = await Promise.all(
voters.map(async (v) => {
const { output } = await generateText({
model: v.model,
instructions: v.instructions,
prompt: task,
output: Output.object({ schema: voteSchema }),
});
ledger.push({ task, voter: v.name, answer: output.answer });
return { voter: v.name, weight: v.weight, ...output };
}),
);
// the fan-in: one round only, a tie is reported rather than rerun
const { output: panel } = await generateText({
model: anthropic('claude-sonnet-5-5'),
instructions: 'Count the weighted votes, report the winner and the split. On a tie, say so and give ' +
'the safer choice. Quote the strongest dissenting reason.',
prompt: `Task: ${task}\n\nBallots:\n` +
votes.map((v) => `- ${v.voter} (weight ${v.weight}): ${v.answer}. ${v.reason}`).join('\n'),
output: Output.object({
schema: z.object({ winner: z.enum(LABELS), split: z.string(), tie: z.boolean(), dissent: z.string() }),
}),
});
return { panel, votes };
}
Give the second voter another provider's model, and replace the equal weights with each voter's measured accuracy from the ledger.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- Voters that differ only in tone are not independent. Vary the method, the starting point or the model.
- Showing any voter another's answer turns the vote into an echo. Run them in parallel and in isolation.
- An unweighted vote lets a weak voter cancel a strong one. Track accuracy per voter and weight once you have the data.
Chain it
Its result can feed
- Chain of Verification conditional
Condition: Build the link only when the task asks a factual question whose short answer can be checked against facts, such as a date, quantity or yes/no fact, rather than a judgment, label or choice between options.
- Debate conditional
Condition: Build the link only when the ensemble's vote was not unanimous, so the payload carries a strongest dissenting argument.
- Textual Backpropagation direct
Treat the winning answer, vote breakdown and reasons as an existing result to evaluate and improve through the configured pipeline.