Agentic Architectures

Catalogue / Sampling & Search

Tree of Thoughts

Branch into several next thoughts, keep the best few, go deeper

Shape
tree-search
Model calls
depth × beam × 2 (propose, score); e.g. 3 levels × 3 paths = 18
Latency
high
Complexity
medium

Also known as ToT, Beam search over thoughts

1

What if the first line of reasoning leads into a dead end?

A model reasoning in one line commits to its first idea and defends it to the end. Tree of Thoughts proposes several different next steps at each stage, scores them with a strict rubric, keeps only the best few and builds on those. A weak early choice is dropped before it costs the whole answer.

You may need it if

  • Answers go wrong because of one bad choice made early, such as the wrong framing of a pricing problem.
  • You can tell a promising partial solution from a weak one before it is finished.
  • The problem has a few distinct approaches, and nobody knows in advance which one works.
2

When is searching a tree of thoughts overkill?

Use it when

  • The problem breaks into steps, and each step has several plausible options: a launch plan, a schedule, a puzzle.
  • You can write a rubric that scores a partial solution, not only a finished one.
  • A wrong answer costs much more than a few dozen extra model calls.

Avoid it when

  • The question has one short final answer, and the risk is a random slip rather than a wrong approach.
    Use Self-Consistency instead
  • Each step is an action in a real system that returns feedback, such as running code or browsing.
    Use Language Agent Tree Search instead
  • The result can be checked by running it, such as tests or a calculation. Running it beats guessing which path is better.
    Use Plan–Execute–Verify instead
  • Partial solutions cannot be judged until they are finished. The scores would be noise and the pruning random.

What it costs

Model calls
depth × beam × 2 (propose, score); e.g. 3 levels × 3 paths = 18
Latency
high
Complexity
medium
3

How does the search grow and get pruned?

A loop over depth. The proposer extends each kept path with a few different next thoughts. The evaluator scores every candidate on a rubric. A rule keeps the best few paths, stores them as the frontier and goes one level deeper, until a path is complete or the depth limit is reached.

no completepath, depthleftcomplete pathor depth spentProblem: The task or question, with the constraints a solution must meet.ProblemProposer: For each kept path, writes a few distinct next thoughts that move it forward.ProposerEvaluator: Scores each candidate path on a fixed rubric and marks paths that are complete.EvaluatorKeep best few: Keeps the top paths by score, drops the rest, and counts the depth used.Keep best fewFrontier: The paths currently kept, each with its thoughts so far and its score.FrontierBest path: The highest-scoring complete path, turned into a final answer.Best path
  1. Step 1 of 5

    Several first steps

    The proposer reads the problem and writes three different first moves, for example three ways to frame a pricing change. They must differ in approach, not in wording.

  2. Step 2 of 5

    Each candidate is scored

    The evaluator sees one candidate path at a time with the rubric. It lists defects, then gives a score, and says whether the path already solves the problem.

  3. Step 3 of 5

    Prune to the beam

    A plain rule keeps the best two or three paths and discards the others. The kept paths go into the frontier. Everything else is forgotten.

  4. Step 4 of 5

    One level deeper

    The proposer reads each kept path from the frontier and writes a few next thoughts for it. The same scoring and pruning follow. Depth is counted at every pass.

  5. Step 5 of 5

    The best complete path wins

    When a path is marked complete, or the depth limit is reached, the highest-scoring path leaves as the answer, with the reasoning steps that led to it.

What each part does
Problem
The task or question, with the constraints a solution must meet.
Proposer
For each kept path, writes a few distinct next thoughts that move it forward.
Evaluator
Scores each candidate path on a fixed rubric and marks paths that are complete.
Keep best few
Keeps the top paths by score, drops the rest, and counts the depth used.
Frontier
The paths currently kept, each with its thoughts so far and its score.
Best path
The highest-scoring complete path, turned into a final answer.
  • Frontier → Proposer: paths to extend
  • Proposer → Evaluator: candidate next thoughts
  • Evaluator → Keep best few: score per candidate
  • Keep best few → Frontier: best paths kept
  • Keep best few → Proposer: no complete path, depth left
  • Keep best few → Best path: complete path or depth spent

Where it connects

Entry
problem
TaskQuestion

A problem that can be solved in a few reasoning steps, with the constraints a valid solution must meet. A scoring rubric is optional, the stage writes one when absent.

Exit
answer
Answer

When: A path is marked complete, or the depth limit is reached.

The final answer, the chain of thoughts that produced it, its score, and whether the path was complete or only the best one found.

Working memory for one run
frontier
Options

The paths kept at the current depth, each with its thoughts in order and its latest score. Replaced at every level and discarded when the run ends.

4

What do I ask my assistant to set up?

The agents to launch

Proposer, one per kept path at each depth, at most 3

You receive a problem and a partial path of reasoning. Write 3 next thoughts that each move the path forward in a clearly different way. Each thought is one step, not a full solution. If a thought completes the solution, say so.

Sees: The problem, its constraints and the one path it extends. Not the other paths, not any score.

Evaluator

Judge one candidate path against the rubric. First list concrete defects, such as a broken constraint, an unsupported jump or a dead end. Then give a score from 1 to 10 and state whether the path is a complete solution. Do not extend the path.

Sees: The problem, the rubric and one candidate path. Not the proposer's reasoning, not the scores of other candidates.

A prompt you can paste

Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.

Framework-agnostic prompt
You are the orchestrator of a multi-agent workflow that follows the "Tree of Thoughts" architecture.
Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it.

# Task
[Describe your task here]

# Tree of Thoughts
Branch into several next thoughts, keep the best few, go deeper.
Input: the task above. It must provide: A problem that can be solved in a few reasoning steps, with the constraints a valid solution must meet. A scoring rubric is optional, the stage writes one when absent. If something essential is missing, ask for it before starting.

Agents to launch:
- Proposer (one per kept path at each depth, at most 3): You receive a problem and a partial path of reasoning. Write 3 next thoughts that each move the path forward in a clearly different way. Each thought is one step, not a full solution. If a thought completes the solution, say so.
  Context for this agent: The problem, its constraints and the one path it extends. Not the other paths, not any score.
- Evaluator: Judge one candidate path against the rubric. First list concrete defects, such as a broken constraint, an unsupported jump or a dead end. Then give a score from 1 to 10 and state whether the path is a complete solution. Do not extend the path.
  Context for this agent: The problem, the rubric and one candidate path. Not the proposer's reasoning, not the scores of other candidates.

Procedure:
1. Restate the problem and its constraints. Write a rubric of three to five checks that can be applied to a partial path.
2. Launch the proposer on the empty path to get 3 distinct first thoughts.
3. Launch the evaluator in a fresh context on each candidate path, one at a time.
4. Keep the 3 highest-scoring paths in the frontier and drop the others. Discard any path that breaks a hard constraint, whatever its score.
5. If no kept path is complete and fewer than 4 levels were used, launch the proposer on each kept path and repeat from the evaluator step.
6. Take the highest-scoring complete path, or the highest-scoring path if none is complete, and write the final answer from it.

Stop rule: Stop as soon as a kept path is marked complete and scores 8 or more, or after 4 levels, whichever comes first.

Exit:
- `answer`, when: A path is marked complete, or the depth limit is reached. Deliver: The final answer, the chain of thoughts that produced it, its score, and whether the path was complete or only the best one found.

# Final deliverable
The final answer, the path of thoughts behind it with the score at each level, and one line stating whether the path was complete or only the best partial one.

The same thing in code

LangGraph, python, package langgraph
from typing import TypedDict, cast

from langchain.chat_models import init_chat_model
from langchain_core.language_models import LanguageModelInput
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel

llm = init_chat_model("anthropic:claude-sonnet-5-5")
BEAM, BRANCH, MAX_DEPTH, GOOD_ENOUGH = 3, 3, 4, 8  # fixed before the run
RUBRIC = ("1. Respects every stated constraint. 2. Each thought follows from the one before. "
          "3. Moves toward a full solution.")  # write three to five checks for your problem

class Thoughts(BaseModel):
    thoughts: list[str]  # BRANCH next steps, each a clearly different approach

class Judgement(BaseModel):
    defects: list[str]
    breaks_constraint: bool
    score: int  # 1 to 10
    complete: bool

class Path(BaseModel):
    thoughts: list[str]
    score: int = 0
    complete: bool = False

class Input(TypedDict):
    problem: str

class State(Input):
    frontier: list[Path]  # the kept paths
    candidates: list[list[str]]
    depth: int
    answer: str

def show(thoughts: list[str]) -> str:
    return "\n".join(f"{i}. {t}" for i, t in enumerate(thoughts, 1)) or "(empty path)"

def seed(state: State) -> dict:
    return {"frontier": [Path(thoughts=[])], "depth": 0}

async def propose(state: State) -> dict:
    # one proposer per kept path; it sees that path only, never a score
    prompts: list[LanguageModelInput] = [
        f"Problem: {state['problem']}\nPath so far:\n{show(p.thoughts)}\nWrite {BRANCH} next thoughts "
        "that each move the path forward in a clearly different way. One step each, not a solution."
        for p in state["frontier"]]
    batches = cast(list[Thoughts], await llm.with_structured_output(Thoughts).abatch(prompts))
    candidates = [p.thoughts + [t] for p, b in zip(state["frontier"], batches) for t in b.thoughts[:BRANCH]]
    return {"candidates": candidates, "depth": state["depth"] + 1}

async def evaluate(state: State) -> dict:
    # a fresh call per candidate: the problem, the rubric and the path, nothing else
    prompts: list[LanguageModelInput] = [
        f"Problem: {state['problem']}\nRubric: {RUBRIC}\nPath:\n{show(c)}\nList concrete defects, then "
        "score 1 to 10 and say whether the path is a complete solution. Do not extend it."
        for c in state["candidates"]]
    judged = cast(list[Judgement], await llm.with_structured_output(Judgement).abatch(prompts))
    # the beam: drop paths that break a hard constraint, keep the best three
    kept = sorted((Path(thoughts=c, score=j.score, complete=j.complete)
                   for c, j in zip(state["candidates"], judged) if not j.breaks_constraint),
                  key=lambda p: p.score, reverse=True)[:BEAM]
    return {"frontier": kept}

def gate(state: State) -> str:
    solved = any(p.complete and p.score >= GOOD_ENOUGH for p in state["frontier"])
    return "finish" if solved or state["depth"] >= MAX_DEPTH or not state["frontier"] else "propose"

async def finish(state: State) -> dict:
    if not state["frontier"]:
        return {"answer": "No path survived the hard constraints."}
    complete = [p for p in state["frontier"] if p.complete]
    best = max(complete or state["frontier"], key=lambda p: p.score)
    reply = await llm.ainvoke(
        f"Problem: {state['problem']}\nWrite the final answer from this path:\n{show(best.thoughts)}")
    label = "complete" if best.complete else "best partial"
    return {"answer": f"{reply.text}\n\nPath ({label}, score {best.score}):\n{show(best.thoughts)}"}

graph = StateGraph(State, input_schema=Input)
graph.add_node("seed", seed)
graph.add_node("propose", propose)
graph.add_node("evaluate", evaluate)
graph.add_node("finish", finish)
graph.add_edge(START, "seed")
graph.add_edge("seed", "propose")
graph.add_edge("propose", "evaluate")
graph.add_conditional_edges("evaluate", gate)
graph.add_edge("finish", END)
app = graph.compile()

async def run(task: str):
    final = await app.ainvoke({"problem": task})
    return final["answer"]

Write a rubric for your own problem and add a near-duplicate filter before the beam, so the three kept paths stay genuinely different.

Claude Agent SDK, python, package claude-agent-sdk
import asyncio

from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
from pydantic import BaseModel

BEAM, BRANCH, MAX_DEPTH, GOOD_ENOUGH = 3, 3, 4, 8  # fixed before the run
RUBRIC = ("1. Respects every stated constraint. 2. Each thought follows from the one before. "
          "3. Moves toward a full solution.")  # write three to five checks for your problem

class Thoughts(BaseModel):
    thoughts: list[str]

class Judgement(BaseModel):
    defects: list[str]
    breaks_constraint: bool
    score: int  # 1 to 10
    complete: bool

AGENTS = {
    "proposer": AgentDefinition(
        description="Extends one partial reasoning path with distinct next thoughts.",
        prompt=f"You receive a problem and a partial path. Write {BRANCH} next thoughts that each move "
        "the path forward in a clearly different way. Each is one step, not a full solution.",
        tools=[]),
    "evaluator": AgentDefinition(
        description="Scores one candidate reasoning path against a rubric.",
        prompt="Judge one candidate path against the rubric. First list concrete defects, then score "
        "it from 1 to 10 and state whether it is a complete solution. Do not extend the path.",
        tools=[]),
}

async def ask(prompt: str, schema: type[BaseModel] | None = None) -> ResultMessage:
    # one query per call: a new session, so a sub-agent never sees other paths or scores
    options = ClaudeAgentOptions(
        agents=AGENTS, allowed_tools=["Agent"],
        output_format={"type": "json_schema", "schema": schema.model_json_schema()} if schema else None)
    async for message in query(prompt=prompt, options=options):
        if isinstance(message, ResultMessage):
            return message
    raise RuntimeError("no result")

def show(path: list[str]) -> str:
    return "\n".join(f"{i}. {t}" for i, t in enumerate(path, 1)) or "(empty path)"

async def run(task: str):
    root = Judgement(defects=[], breaks_constraint=False, score=0, complete=False)
    frontier: list[tuple[list[str], Judgement]] = [([], root)]  # the kept paths
    depth = 0
    while depth < MAX_DEPTH:  # at most 4 levels
        depth += 1
        proposals = await asyncio.gather(*(ask(
            f"Delegate to the proposer agent.\nProblem: {task}\nPath so far:\n{show(p)}", Thoughts)
            for p, _ in frontier))
        candidates = [p + [t] for (p, _), m in zip(frontier, proposals)
                      for t in Thoughts.model_validate(m.structured_output).thoughts[:BRANCH]]
        judged = await asyncio.gather(*(ask(
            f"Delegate to the evaluator agent.\nProblem: {task}\nRubric: {RUBRIC}\nPath:\n{show(c)}",
            Judgement) for c in candidates))
        scored = [(c, Judgement.model_validate(m.structured_output)) for c, m in zip(candidates, judged)]
        # the beam: drop paths that break a hard constraint, keep the best three
        frontier = sorted([(c, j) for c, j in scored if not j.breaks_constraint],
                          key=lambda cj: cj[1].score, reverse=True)[:BEAM]
        if not frontier or any(j.complete and j.score >= GOOD_ENOUGH for _, j in frontier):
            break
    if not frontier:
        return {"answer": None, "reason": "no path survived the hard constraints"}
    complete = [(c, j) for c, j in frontier if j.complete]
    path, best = max(complete or frontier, key=lambda cj: cj[1].score)
    final = await ask(f"Problem: {task}\nWrite the final answer from this path:\n{show(path)}")
    return {"answer": final.result, "path": path, "score": best.score,
            "complete": best.complete, "depth": depth}

Write a rubric for your own problem and reject near-duplicate thoughts before scoring; calls grow as beam times branch times depth.

OpenAI Agents SDK, python, package openai-agents
import asyncio

from agents import Agent, Runner
from pydantic import BaseModel

BEAM, BRANCH, MAX_DEPTH, GOOD_ENOUGH = 3, 3, 4, 8  # fixed before the run
RUBRIC = ("1. Respects every stated constraint. 2. Each thought follows from the one before. "
          "3. Moves toward a full solution.")  # write three to five checks for your problem

class Thoughts(BaseModel):
    thoughts: list[str]

class Judgement(BaseModel):
    defects: list[str]
    breaks_constraint: bool
    score: int  # 1 to 10
    complete: bool

proposer = Agent(
    name="Proposer",
    instructions=f"You receive a problem and a partial path. Write {BRANCH} next thoughts that each "
    "move the path forward in a clearly different way. Each is one step, not a full solution.",
    output_type=Thoughts,
)
evaluator = Agent(
    name="Evaluator",
    instructions="Judge one candidate path against the rubric. First list concrete defects, then "
    "score it from 1 to 10 and state whether it is a complete solution. Do not extend the path.",
    output_type=Judgement,
)
writer = Agent(name="Writer", instructions="Write the final answer from the given path of thoughts.")

def show(path: list[str]) -> str:
    return "\n".join(f"{i}. {t}" for i, t in enumerate(path, 1)) or "(empty path)"

async def run(task: str):
    root = Judgement(defects=[], breaks_constraint=False, score=0, complete=False)
    frontier: list[tuple[list[str], Judgement]] = [([], root)]  # the kept paths
    depth = 0
    while depth < MAX_DEPTH:  # at most 4 levels
        depth += 1
        # one run per kept path: a proposer sees its own path, never a score
        proposals = await asyncio.gather(*(
            Runner.run(proposer, f"Problem: {task}\nPath so far:\n{show(p)}") for p, _ in frontier))
        candidates = [p + [t] for (p, _), r in zip(frontier, proposals)
                      for t in r.final_output_as(Thoughts).thoughts[:BRANCH]]
        judged = await asyncio.gather(*(
            Runner.run(evaluator, f"Problem: {task}\nRubric: {RUBRIC}\nPath:\n{show(c)}")
            for c in candidates))
        scored = [(c, r.final_output_as(Judgement)) for c, r in zip(candidates, judged)]
        # the beam: drop paths that break a hard constraint, keep the best three
        frontier = sorted([(c, j) for c, j in scored if not j.breaks_constraint],
                          key=lambda cj: cj[1].score, reverse=True)[:BEAM]
        if not frontier or any(j.complete and j.score >= GOOD_ENOUGH for _, j in frontier):
            break
    if not frontier:
        return {"answer": None, "reason": "no path survived the hard constraints"}
    complete = [(c, j) for c, j in frontier if j.complete]
    path, best = max(complete or frontier, key=lambda cj: cj[1].score)
    final = await Runner.run(writer, f"Problem: {task}\nPath:\n{show(path)}")
    return {"answer": final.final_output, "path": path, "score": best.score,
            "complete": best.complete, "depth": depth}

Write a rubric for your own problem and reject near-duplicate thoughts before scoring; calls grow as beam times branch times depth.

Vercel AI SDK, typescript, package ai
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';

const model = anthropic('claude-sonnet-5-5');
const [BEAM, BRANCH, MAX_DEPTH, GOOD_ENOUGH] = [3, 3, 4, 8]; // fixed before the run
// write three to five checks for your problem
const RUBRIC =
  '1. Respects every stated constraint. 2. Each thought follows from the one before. ' +
  '3. Moves toward a full solution.';

const thoughtsSchema = z.object({ thoughts: z.array(z.string()) });
const judgementSchema = z.object({
  defects: z.array(z.string()),
  breaksConstraint: z.boolean(),
  score: z.number().int().min(1).max(10),
  complete: z.boolean(),
});
type Scored = { path: string[]; score: number; complete: boolean };
const show = (path: string[]) => path.map((t, i) => `${i + 1}. ${t}`).join('\n') || '(empty path)';

export async function run(task: string) {
  let frontier: Scored[] = [{ path: [], score: 0, complete: false }];
  let depth = 0;
  while (depth < MAX_DEPTH) { // at most 4 levels
    depth++;
    // one call per kept path: a proposer sees its own path, never a score
    const proposals = await Promise.all(frontier.map(async ({ path }) => {
      const { output } = await generateText({
        model,
        instructions:
          `Write ${BRANCH} next thoughts that each move the path forward in a clearly ` +
          'different way. Each is one step, not a full solution.',
        prompt: `Problem: ${task}\nPath so far:\n${show(path)}`,
        output: Output.object({ schema: thoughtsSchema }),
      });
      return output.thoughts.slice(0, BRANCH).map((t) => [...path, t]);
    }));
    // a fresh call per candidate: problem, rubric and path only
    const scored = await Promise.all(proposals.flat().map(async (path) => {
      const { output } = await generateText({
        model,
        instructions:
          'Judge the path against the rubric. List concrete defects first, then score 1 to 10 ' +
          'and say whether it is a complete solution. Do not extend it.',
        prompt: `Problem: ${task}\nRubric: ${RUBRIC}\nPath:\n${show(path)}`,
        output: Output.object({ schema: judgementSchema }),
      });
      return { path, ...output };
    }));
    // the beam: drop paths that break a hard constraint, keep the best three
    frontier = scored.filter((s) => !s.breaksConstraint).sort((a, b) => b.score - a.score).slice(0, BEAM);
    if (frontier.length === 0 || frontier.some((s) => s.complete && s.score >= GOOD_ENOUGH)) break;
  }
  if (frontier.length === 0) return { answer: null, reason: 'no path survived the hard constraints' };
  const best = frontier.find((s) => s.complete) ?? frontier[0]; // frontier is sorted by score
  const { text: answer } = await generateText({
    model,
    prompt: `Problem: ${task}\nWrite the final answer from this path:\n${show(best.path)}`,
  });
  return { answer, path: best.path, score: best.score, complete: best.complete, depth };
}

Write a rubric for your own problem and reject near-duplicate thoughts before scoring; calls grow as beam times branch times depth.

Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.

What goes wrong

  • Candidates that differ only in wording waste the beam. Ask for different approaches and reject near-duplicates.
  • An evaluator that also sees the proposer's justification scores the argument, not the path. Keep it to the path itself.
  • The calls multiply with beam and depth. Fix both numbers before the run, not during it.

Chain it

Its result can feed

Source