1Why does the same prompt sometimes give a wrong answer it would not give twice?
A model that reasons step by step can slip on one step and land on a wrong answer. Slips are random, so they rarely repeat. Self-consistency asks the same prompt several times with some randomness, compares only the final answers and keeps the most frequent one. Sound reasoning converges, slips scatter.
You may need it if
- The same question, asked twice, sometimes gets two different answers.
- The answer is short and comparable, such as a number, a date, a category or yes or no.
- Your prompt is right most of the time, and the errors look random rather than systematic.
2When does voting over samples not help?
Use it when
- The question has one checkable final answer: an invoice total, an eligibility decision, a ticket category.
- The prompt is already good and the remaining errors come from occasional slips.
- Accuracy matters more than paying five to ten times for each answer.
Avoid it when
- The prompt is wrong in a consistent way, such as a misread policy. Every sample repeats the error and the vote confirms it.
Use Ensemble instead - The answer is a long text, such as an email or a proposal. There is no majority among texts that all differ.
Use Reflection instead - The answer rests on facts the model does not have. Samples will agree on a guess.
Use Corrective RAG instead - Errors come from a wrong approach chosen early, not from slips along the way.
Use Tree of Thoughts instead
What it costs
- Model calls
- N samples + 1 normalisation (N = 5, topped up to 10 if needed)
- Latency
- low
- Complexity
- low
3How do the samples become one answer?
A fan-out and a count. The same prompt runs N times in parallel with some randomness. A normaliser reduces each run to its final answer in a standard form. A plain count picks the majority, and a weak majority triggers one top-up batch of samples.
- Step 1 of 5
Same prompt, several runs
The question, for example "which refund rule applies to this order?", is sent five times with the same prompt and some randomness. Every run reasons in its own way and ends with an answer.
- Step 2 of 5
Only the final answers count
The normaliser strips the reasoning and writes each final answer in one standard form. The reasoning is ignored on purpose, so a convincing argument gets one vote like any other.
- Step 3 of 5
Count
A plain count, not a model, gives each answer's share. If the top answer holds at least 60% of the votes, it wins.
- Step 4 of 5
One top-up when unsure
Below the threshold, five more samples are drawn once and counted with the first ones. There is no second top-up.
- Step 5 of 5
The majority and its share
The winning answer leaves with its share of votes. A reader can tell an answer that won 9 to 1 from one that scraped through.
What each part does
- Question
- The question or task, with the form the final answer must take.
- Samplers
- The same prompt, run N times with some randomness. Each run reasons and ends with a final answer.
- Normaliser
- Reduces each run to its final answer in a standard form, so that 1,200 and 1200.00 count as one.
- Clear majority?
- Counts the normalised answers and compares the top share with the threshold.
- Majority answer
- The most frequent answer, with its share of the votes and one reasoning that reached it.
- Samplers → Normaliser: N reasonings
- Normaliser → Clear majority?: N final answers
- Clear majority? → Samplers: below threshold, no top-up yet
- Clear majority? → Majority answer: majority found or top-up spent
Where it connects
A question or task with one short final answer that can be compared across runs. If the answer form is not stated, the stage fixes one.
When: The top answer reaches the threshold, or the single top-up batch has been counted.
The majority answer, its share of the votes, the full tally, one reasoning that reached it, and a flag when the share stayed below the threshold.
4What do I ask my assistant to set up?
The agents to launch
Sampler, 5 in parallel, plus 5 more once if the majority is weak
Answer the question. Reason step by step, then end with one line that starts with "Final answer:" followed by the answer in the required form. Use the same instructions for every run, with moderate randomness, not the most predictable setting.
Sees: The question and the required answer form. Never another run's output.
Normaliser
For each run, extract the final answer and rewrite it in the standard form: same units, same spelling of categories, numbers without formatting. If two answers mean the same thing, write them identically. If a run has no final answer, mark it as invalid. Do not judge which answer is right.
Sees: The required answer form and the final lines of every run. Not the reasoning that led to them.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Self-Consistency" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Self-Consistency Ask the same question several times and keep the majority answer. Input: the task above. It must provide: A question or task with one short final answer that can be compared across runs. If the answer form is not stated, the stage fixes one. If something essential is missing, ask for it before starting. Agents to launch: - Sampler (5 in parallel, plus 5 more once if the majority is weak): Answer the question. Reason step by step, then end with one line that starts with "Final answer:" followed by the answer in the required form. Use the same instructions for every run, with moderate randomness, not the most predictable setting. Context for this agent: The question and the required answer form. Never another run's output. - Normaliser: For each run, extract the final answer and rewrite it in the standard form: same units, same spelling of categories, numbers without formatting. If two answers mean the same thing, write them identically. If a run has no final answer, mark it as invalid. Do not judge which answer is right. Context for this agent: The required answer form and the final lines of every run. Not the reasoning that led to them. Procedure: 1. Fix the form of the final answer, such as one category from a list, a number with its unit, or yes or no. 2. Launch 5 samplers at the same time, each in a fresh context, with the same prompt and moderate randomness. 3. Launch the normaliser on the final lines of all runs. 4. Count the normalised answers, leaving out invalid ones. If the top answer has at least 60% of the valid votes, it wins. 5. If it does not and no top-up was drawn, launch 5 more samplers, normalise their answers and count all 10 together. Stop rule: Stop when the top answer holds 60% of the valid votes, or after one top-up of 5 samples. Never draw a third batch. Exit: - `answer`, when: The top answer reaches the threshold, or the single top-up batch has been counted. Deliver: The majority answer, its share of the votes, the full tally, one reasoning that reached it, and a flag when the share stayed below the threshold. # Final deliverable The majority answer, the full tally, its share of the votes, one reasoning that reached it, and a clear flag if the share stayed below 60%.
The same thing in code
import operator
from collections import Counter
from typing import Annotated, TypedDict
from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from langgraph.types import Send
from pydantic import BaseModel
# moderate randomness for the samplers, so that slips scatter instead of repeating
sampler_llm = init_chat_model("anthropic:claude-sonnet-5-5", temperature=0.8)
normaliser_llm = init_chat_model("anthropic:claude-sonnet-5-5", temperature=0)
SAMPLER = ("Answer the question. Reason step by step, then end with one line that starts with "
"'Final answer:' followed by the answer in the required form.")
NORMALISER = ("For each final line, rewrite the answer in the standard form: same units, same spelling "
"of categories, numbers without formatting. Write INVALID when a run has no final answer. "
"Do not judge which answer is right.")
BATCH, THRESHOLD, MAX_BATCHES = 5, 0.6, 2
class Normalised(BaseModel):
answers: list[str] # one per run, same order
class State(TypedDict):
task: str
runs: Annotated[list[str], operator.add]
answers: list[str]
batches: int
winner: str
share: float
class SamplerInput(TypedDict):
task: str
def launch_batch(state: State) -> list[Send]:
return [Send("sampler", {"task": state["task"]}) for _ in range(BATCH)]
async def sampler(state: SamplerInput) -> dict:
# each run gets only the question, never another run's output
msg = await sampler_llm.ainvoke([("system", SAMPLER), ("user", state["task"])])
return {"runs": [msg.text]}
async def normaliser(state: State) -> dict:
# the normaliser sees the final lines only, not the reasoning
finals = [r.rsplit("Final answer:", 1)[1].strip() if "Final answer:" in r else "" for r in state["runs"]]
lines = "\n".join(f"{i}. {f}" for i, f in enumerate(finals))
out = await normaliser_llm.with_structured_output(Normalised).ainvoke(
[("system", NORMALISER), ("user", f"Question and answer form: {state['task']}\nFinal lines:\n{lines}")])
return {"answers": Normalised.model_validate(out).answers, "batches": state["batches"] + 1}
def tally(state: State) -> dict:
# a plain count, not a model
valid = [a for a in state["answers"] if a != "INVALID"]
if not valid:
return {"winner": "", "share": 0.0}
winner, votes = Counter(valid).most_common(1)[0]
return {"winner": winner, "share": votes / len(valid)}
def clear_majority(state: State) -> list[Send] | str:
# the gate: 60% of valid votes, or stop after one top-up batch
if state["share"] >= THRESHOLD or state["batches"] >= MAX_BATCHES:
return END
return launch_batch(state)
builder = StateGraph(State)
builder.add_node("sampler", sampler)
builder.add_node("normaliser", normaliser)
builder.add_node("tally", tally)
builder.add_conditional_edges(START, launch_batch, ["sampler"])
builder.add_edge("sampler", "normaliser")
builder.add_edge("normaliser", "tally")
builder.add_conditional_edges("tally", clear_majority, ["sampler", END])
graph = builder.compile()
async def run(task: str):
s = await graph.ainvoke({"task": task, "runs": [], "answers": [], "batches": 0, "winner": "", "share": 0.0})
pairs = list(zip(s["runs"], s["answers"]))
return {
"answer": s["winner"],
"share": s["share"],
"tally": Counter(a for _, a in pairs if a != "INVALID"),
"reasoning": next((r for r, a in pairs if a == s["winner"]), ""),
"weak_majority": s["share"] < THRESHOLD,
}
Calibrate the 60% threshold and the sampler temperature on a labelled set of your own questions before trusting the vote.
from collections import Counter
from typing import Any
from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
BATCH, THRESHOLD, MAX_BATCHES = 5, 0.6, 2
AGENTS = {
"sampler": AgentDefinition(
description="Answers the question once, reasoning step by step.",
prompt=("Answer the question. Reason step by step, then end with one line that starts with "
"'Final answer:' followed by the answer in the required form."),
tools=[],
),
"normaliser": AgentDefinition(
description="Rewrites a list of final answers in one standard form.",
prompt=("For each final line, rewrite the answer in the standard form: same units, same spelling, "
"numbers without formatting. Write INVALID when a line has no answer. Do not judge which is right."),
tools=[],
),
}
BATCH_SCHEMA = {"type": "json_schema", "schema": {
"type": "object", "required": ["runs"],
"properties": {"runs": {"type": "array", "items": {
"type": "object", "required": ["reasoning", "answer"],
"properties": {"reasoning": {"type": "string"}, "answer": {"type": "string"}}}}}}}
OPTIONS = ClaudeAgentOptions(agents=AGENTS, tools=["Agent"], allowed_tools=["Agent"], output_format=BATCH_SCHEMA)
async def ask(prompt: str) -> Any:
async for message in query(prompt=prompt, options=OPTIONS):
if isinstance(message, ResultMessage):
return message.structured_output
raise RuntimeError("no result")
async def sample_batch(task: str) -> list[dict[str, str]]:
# one batch: N sampler sub-agents in parallel, each in its own context, then the normaliser
out = await ask(
f"Launch the sampler sub-agent {BATCH} times in parallel, giving each only this question:\n{task}\n"
"Then give the normaliser sub-agent the question and only the 'Final answer:' lines, not the reasoning. "
"Return each run's full text as reasoning and its normalised answer.")
return out["runs"]
async def run(task: str):
runs: list[dict[str, str]] = []
winner, share = "", 0.0
for _ in range(MAX_BATCHES): # the gate: one top-up at most, never a third batch
runs += await sample_batch(task)
votes = Counter(r["answer"] for r in runs if r["answer"] != "INVALID")
if votes:
winner, n = votes.most_common(1)[0]
share = n / sum(votes.values())
if share >= THRESHOLD:
break
return {
"answer": winner,
"share": share,
"tally": Counter(r["answer"] for r in runs if r["answer"] != "INVALID"),
"reasoning": next((r["reasoning"] for r in runs if r["answer"] == winner), ""),
"weak_majority": share < THRESHOLD,
}
The Agent SDK sets no temperature, so variety comes only from separate sub-agent contexts; check that runs really differ before trusting the vote.
import asyncio
from collections import Counter
from agents import Agent, ModelSettings, Runner
from pydantic import BaseModel
BATCH, THRESHOLD, MAX_BATCHES = 5, 0.6, 2
class Normalised(BaseModel):
answers: list[str] # one per final line, same order; INVALID when missing
sampler = Agent(
name="Sampler",
instructions=("Answer the question. Reason step by step, then end with one line that starts with "
"'Final answer:' followed by the answer in the required form."),
model_settings=ModelSettings(temperature=0.8), # some randomness, not the most predictable setting
)
normaliser = Agent(
name="Normaliser",
instructions=("For each final line, rewrite the answer in the standard form: same units, same spelling "
"of categories, numbers without formatting. Write INVALID when a line has no answer. "
"Do not judge which answer is right."),
output_type=Normalised,
)
def final_line(text: str) -> str:
return text.rsplit("Final answer:", 1)[1].strip() if "Final answer:" in text else ""
async def sample_batch(task: str) -> list[str]:
# fan-out: separate runs, none sees another's output
results = await asyncio.gather(*(Runner.run(sampler, task) for _ in range(BATCH)))
return [str(r.final_output) for r in results]
async def run(task: str):
runs: list[str] = []
answers: list[str] = []
winner, share = "", 0.0
for _ in range(MAX_BATCHES): # the gate: one top-up at most
runs += await sample_batch(task)
lines = "\n".join(f"{i}. {final_line(r)}" for i, r in enumerate(runs))
result = await Runner.run(normaliser, f"Question and answer form: {task}\nFinal lines:\n{lines}")
answers = result.final_output_as(Normalised).answers
votes = Counter(a for a in answers if a != "INVALID")
if votes:
winner, n = votes.most_common(1)[0]
share = n / sum(votes.values())
if share >= THRESHOLD:
break
return {
"answer": winner,
"share": share,
"tally": Counter(a for a in answers if a != "INVALID"),
"reasoning": next((r for r, a in zip(runs, answers) if a == winner), ""),
"weak_majority": share < THRESHOLD,
}
Confirm that the SDK's default model accepts a temperature setting, or pick one that does, so the five runs really vary.
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';
const model = anthropic('claude-sonnet-5-5');
const BATCH = 5, THRESHOLD = 0.6, MAX_BATCHES = 2;
const SAMPLER =
"Answer the question. Reason step by step, then end with one line that starts with " +
"'Final answer:' followed by the answer in the required form.";
const NORMALISER =
'For each final line, rewrite the answer in the standard form: same units, same spelling of ' +
'categories, numbers without formatting. Write INVALID when a line has no answer. ' +
'Do not judge which answer is right.';
function finalLine(text: string): string {
const i = text.lastIndexOf('Final answer:');
return i < 0 ? '' : text.slice(i + 'Final answer:'.length).trim();
}
async function sampleBatch(task: string): Promise<string[]> {
// fan-out: each run sees only the question; moderate randomness
const runs = await Promise.all(
Array.from({ length: BATCH }, () =>
generateText({ model, instructions: SAMPLER, prompt: task, temperature: 0.8 })),
);
return runs.map((r) => r.text);
}
async function normalise(task: string, runs: string[]): Promise<string[]> {
// the normaliser gets the final lines only, not the reasoning
const lines = runs.map((r, i) => `${i}. ${finalLine(r)}`).join('\n');
const { output } = await generateText({
model,
instructions: NORMALISER,
prompt: `Question and answer form: ${task}\nFinal lines:\n${lines}`,
output: Output.object({ schema: z.object({ answers: z.array(z.string()) }) }),
});
return output.answers;
}
export async function run(task: string) {
let runs: string[] = [];
let answers: string[] = [];
let winner = '', share = 0;
const tally = new Map<string, number>();
for (let batch = 0; batch < MAX_BATCHES; batch++) { // the gate: one top-up at most
runs = runs.concat(await sampleBatch(task));
answers = await normalise(task, runs);
tally.clear();
winner = '';
share = 0;
for (const a of answers) if (a !== 'INVALID') tally.set(a, (tally.get(a) ?? 0) + 1);
const valid = [...tally.values()].reduce((s, n) => s + n, 0);
for (const [a, n] of tally) if (valid && n / valid > share) { winner = a; share = n / valid; }
if (share >= THRESHOLD) break;
}
return {
answer: winner,
share,
tally: Object.fromEntries(tally),
reasoning: runs[answers.indexOf(winner)] ?? '',
weakMajority: share < THRESHOLD,
};
}
Calibrate the 60% threshold and the sampler temperature on a labelled set of your own questions before trusting the vote.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- At the most predictable setting all runs are identical and the vote is empty. Allow some randomness.
- Counting before normalising splits one answer into several spellings and hands the win to a wrong one.
- The vote cannot fix a wrong prompt. If samples agree on wrong answers, test the prompt, or switch to an ensemble.
Chain it
Its result can feed
- Chain of Verification conditional
Condition: Build the link when the task asks for a factual answer (a date, name, figure or attribute), not a computation or logic puzzle.
- Debate direct
Treat the majority answer as the existing answer to challenge, and use the tally's minority answers and any low-share flag as candidate objections.
- Textual Backpropagation direct
Treat the majority answer and its reasoning as an existing result to improve, and use the vote share and the low-majority flag as hints of where it may be wrong.