Agentic Architectures

Catalogue / Sampling & Search

Graph of Thoughts

Split, score and keep the best, then merge branches into one

Shape
tree-search
Model calls
per operation: attempts + scores; e.g. 4 parts × 6, merge 6, refine 4 = 34
Latency
high
Complexity
high

Also known as GoT

1

What if the best result is a combination of several branches?

Each model output is a vertex in a graph, linked to the outputs it was made from. A plan written before the run says when to generate several attempts, score them and keep the best, merge several kept results into one new result, or improve a result in a loop. A tree can only extend and prune separate paths.

You may need it if

  • The work splits into parts, and the parts' results must be combined into one result.
  • Two candidates each get a different half right, and choosing one loses the other half.
  • A single merge of all the parts comes out with duplicates, gaps or contradictions.
  • You already know how the problem decomposes and what a good partial result looks like.
2

When is a graph of thoughts more machinery than the task needs?

Use it when

  • The task decomposes into parts whose results can be merged, such as several sources into one document.
  • You can write the sequence of operations in advance: where to split, how many attempts, when to merge.
  • A partial result can be scored, by a rubric or by code, before the whole is finished.
  • A weak merge costs more than a few dozen extra model calls.

Avoid it when

  • The problem is one line of reasoning where an early choice may be wrong. There is nothing to merge, only paths to extend and prune.
    Use Tree of Thoughts instead
  • Each part is processed once and the results are merged once, with no competing attempts to score.
    Use Map-Reduce instead
  • Nobody knows before the run how the problem splits. The plan of operations here is written by hand in advance.
    Use As-Needed Decomposition instead
  • The question has one short final answer, and the risk is a random slip rather than a poor combination.
    Use Self-Consistency instead
  • Partial results cannot be scored. Keeping the best attempt would then be a random choice.

What it costs

Model calls
per operation: attempts + scores; e.g. 4 parts × 6, merge 6, refine 4 = 34
Latency
high
Complexity
high
3

Who decides when to split, score, merge and refine?

A controller in plain code follows a plan of operations fixed before the run. Each operation takes thoughts from the graph and sends them to a generator, an aggregator or a refiner. A scorer rates every new thought and the best are kept. The thought left when the plan is exhausted is the result.

plan says:generateplan says: mergeplan says:refinescored thoughtsplan exhaustedProblem: The task or question, with the material it applies to and the constraints on the result.ProblemNext operation?: Plain code. Follows the operation plan, picks the thoughts each operation works on and keeps the best-scored ones.Nextoperation?Thought graph: Every thought produced so far, with its score and links to the thoughts it was made from.Thought graphGenerator: Writes several new thoughts from one: the parts of a task, or several attempts at one part.GeneratorAggregator: Writes one new thought from several kept thoughts. This is the step a tree does not have.AggregatorRefiner: Rewrites one thought to remove the defects found by the scorer, without changing what it covers.RefinerScorer: Rates each new thought against a fixed rubric and lists its defects. Can be plain code when the task allows.ScorerResult: The thought kept after the last operation, with its score and the graph that led to it.Result
  1. Step 1 of 6

    A plan written before the run

    A product team has four long interview transcripts and wants one list of the customers' main problems. The plan is fixed in advance: one part per transcript, three attempts per part, keep the best, merge, then refine twice at most.

  2. Step 2 of 6

    Several attempts per part

    The first operation is a generation. For each transcript, three generators each write a list of the problems it mentions. They work apart, so the three lists differ in what they catch.

  3. Step 3 of 6

    Score, and keep the best

    The scorer rates each list against the rubric: every problem traced to a quote, nothing invented, no duplicates. The controller records all twelve lists in the graph and keeps the best one for each transcript.

  4. Step 4 of 6

    Several thoughts in, one out

    The plan now says merge. The aggregator receives the four kept lists and writes a single list from them. The new thought has four incoming links. It is attempted three times, and the scorer picks the best merge.

  5. Step 5 of 6

    Refine in a loop

    The best merge scores 6: two problems are stated twice in different words. The refiner rewrites it with those defects in hand. The scorer rates the new version, and the controller keeps whichever of the two scores higher.

  6. Step 6 of 6

    The plan is exhausted

    The refined list scores 8, so the second refinement is skipped. No operation is left. The kept thought leaves as the result, with the graph showing which transcript lists it was built from.

What each part does
Problem
The task or question, with the material it applies to and the constraints on the result.
Next operation?
Plain code. Follows the operation plan, picks the thoughts each operation works on and keeps the best-scored ones.
Thought graph
Every thought produced so far, with its score and links to the thoughts it was made from.
Generator
Writes several new thoughts from one: the parts of a task, or several attempts at one part.
Aggregator
Writes one new thought from several kept thoughts. This is the step a tree does not have.
Refiner
Rewrites one thought to remove the defects found by the scorer, without changing what it covers.
Scorer
Rates each new thought against a fixed rubric and lists its defects. Can be plain code when the task allows.
Result
The thought kept after the last operation, with its score and the graph that led to it.
  • Thought graph → Next operation?: thoughts to work on
  • Next operation? → Generator: plan says: generate
  • Next operation? → Aggregator: plan says: merge
  • Next operation? → Refiner: plan says: refine
  • Scorer → Next operation?: scored thoughts
  • Next operation? → Thought graph: vertices, links, scores
  • Next operation? → Result: plan exhausted

Where it connects

Entry
problem
TaskQuestion

A problem that splits into parts whose results can be merged, with its constraints. The plan of operations and the scoring rubric are already configured.

Exit
answer
Answer

When: The last operation of the plan has run, or the refined thought reached the target score.

The final thought as the answer, its score, and the list of thoughts with their scores and the thoughts each one was made from.

Working memory for one run
thoughts
Options

All thoughts of the run as vertices, each with its text, its score and links to its source thoughts. Dropped attempts stay recorded. Discarded when the run ends.

4

What do I ask my assistant to set up?

The agents to launch

Generator, 3 per part, each in a fresh context; one more to split the task into parts

You receive a task and one part of it. Produce a complete result for that part only. Do not cover the other parts and do not refer to them. Work from the material given and add nothing that is not in it.

Sees: The task and its one part with the material for it. Not the other parts, not the other attempts, not any score.

Aggregator, 3 merge attempts, each in a fresh context

You receive several partial results for one task. Write a single result that keeps everything they contain, states each item once and resolves contradictions explicitly. Do not add content that is in none of them.

Sees: The task and the kept partial results. Not the attempts that were dropped, not the scores.

Refiner

You receive one result and the list of defects a reviewer found in it. Rewrite the result so that each defect is fixed. Keep everything that was not criticised. Do not shorten it and do not add new content.

Sees: The task, the one result to improve and its list of defects. Not the earlier attempts.

Scorer

Judge one result against the rubric. First list concrete defects, such as a missing item, a duplicate, an invented fact or a broken constraint. Then give a score from 1 to 10. Do not rewrite the result.

Sees: The goal of the part or of the whole task, the rubric and one result. Not who wrote it, not the other results.

A prompt you can paste

Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.

Framework-agnostic prompt
You are the orchestrator of a multi-agent workflow that follows the "Graph of Thoughts" architecture.
Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it.

# Task
[Describe your task here]

# Graph of Thoughts
Split, score and keep the best, then merge branches into one.
Input: the task above. It must provide: A problem that splits into parts whose results can be merged, with its constraints. The plan of operations and the scoring rubric are already configured. If something essential is missing, ask for it before starting.

Agents to launch:
- Generator (3 per part, each in a fresh context; one more to split the task into parts): You receive a task and one part of it. Produce a complete result for that part only. Do not cover the other parts and do not refer to them. Work from the material given and add nothing that is not in it.
  Context for this agent: The task and its one part with the material for it. Not the other parts, not the other attempts, not any score.
- Aggregator (3 merge attempts, each in a fresh context): You receive several partial results for one task. Write a single result that keeps everything they contain, states each item once and resolves contradictions explicitly. Do not add content that is in none of them.
  Context for this agent: The task and the kept partial results. Not the attempts that were dropped, not the scores.
- Refiner: You receive one result and the list of defects a reviewer found in it. Rewrite the result so that each defect is fixed. Keep everything that was not criticised. Do not shorten it and do not add new content.
  Context for this agent: The task, the one result to improve and its list of defects. Not the earlier attempts.
- Scorer: Judge one result against the rubric. First list concrete defects, such as a missing item, a duplicate, an invented fact or a broken constraint. Then give a score from 1 to 10. Do not rewrite the result.
  Context for this agent: The goal of the part or of the whole task, the rubric and one result. Not who wrote it, not the other results.

Procedure:
1. Write the plan of operations before anything runs: how the task splits into 2 to 4 parts, 3 attempts per part, 3 merge attempts, at most 2 refinements. Do not change it during the run.
2. Write a rubric of three to five checks that apply to a partial result and to the merged one.
3. For each part, launch 3 generators in separate contexts. Launch the scorer on each attempt, one at a time. Keep the best attempt of each part.
4. Record every attempt in the thought graph with its score and the thought it came from, including the ones you drop.
5. Launch the aggregator 3 times on the kept attempts. Score the 3 merges and keep the best. Record it with links to all of its sources.
6. If the kept merge scores under 8, launch the refiner with the scorer's defects, score the new version and keep the higher of the two. Do this at most twice.
7. Return the kept thought when no operation is left.

Stop rule: Stop when the plan is exhausted. With 4 parts that is 12 attempts, 3 merges and at most 2 refinements, each scored once. Refinement ends early when the kept thought scores 8 or more.

Exit:
- `answer`, when: The last operation of the plan has run, or the refined thought reached the target score. Deliver: The final thought as the answer, its score, and the list of thoughts with their scores and the thoughts each one was made from.

# Final deliverable
The final result and its score, then the thought graph as a list. For each thought, the operation that made it, its score and the thoughts it was made from.

The same thing in code

LangGraph, python, package langgraph
import asyncio
from typing import TypedDict

from langchain.chat_models import init_chat_model
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel

# the plan of operations, fixed before the run: split, attempt each part, merge, refine
MAX_PARTS, ATTEMPTS, MERGES, MAX_REFINES, GOOD_ENOUGH = 4, 3, 3, 2, 8
RUBRIC = "1. Covers everything in the material. 2. States each item once. 3. Adds nothing that is not in the material."
GENERATOR = "Produce a complete result for your part only. Work from the material given and add nothing to it."
AGGREGATOR = ("Write a single result from the partial results. Keep everything they contain, state each item "
              "once and resolve contradictions explicitly. Add nothing that is in none of them.")
REFINER = "Rewrite the result so that each listed defect is fixed. Keep everything else. Add no new content."
SCORER = "Judge the result against the rubric. List concrete defects first, then score 1 to 10. Do not rewrite it."

llm = init_chat_model("anthropic:claude-sonnet-5-5")


class Parts(BaseModel):
    parts: list[str]  # independent parts whose results can be merged


class Score(BaseModel):
    defects: list[str]
    score: int  # 1 to 10


class Thought(BaseModel):  # a vertex; parents are its incoming links
    id: int
    op: str
    text: str
    parents: list[int]
    score: int = 0
    defects: list[str] = []


class State(TypedDict):
    task: str
    graph: list[Thought]  # every thought of the run, kept or dropped
    kept: list[int]  # ids the next operation works on
    refines: int


async def scored(next_id: int, goal: str, texts: list[str], parents: list[int], op: str) -> list[Thought]:
    # a fresh scorer call per thought: the goal, the rubric and the text, nothing else
    judge = llm.with_structured_output(Score)
    raw = await asyncio.gather(*(judge.ainvoke(f"{SCORER}\nGoal: {goal}\nRubric: {RUBRIC}\n\nResult:\n{t}") for t in texts))
    new = []
    for text, r in zip(texts, raw):
        s = Score.model_validate(r)
        new.append(Thought(id=next_id + len(new), op=op, text=text, parents=parents, score=s.score, defects=s.defects))
    return new


async def write(instructions: str, prompt: str) -> str:
    return (await llm.ainvoke(f"{instructions}\n\n{prompt}")).text


async def generate(state: State) -> dict:
    task = state["task"]
    split = Parts.model_validate(await llm.with_structured_output(Parts).ainvoke(
        f"Split this task into 2 to {MAX_PARTS} independent parts whose results can be merged.\n\n{task}"))
    graph, kept = list(state["graph"]), []
    for part_text in split.parts[:MAX_PARTS]:
        part = Thought(id=len(graph), op="part", text=part_text, parents=[])
        graph.append(part)
        # several attempts per part, each in a fresh context; the best one survives
        texts = await asyncio.gather(*(write(GENERATOR, f"Task: {task}\nYour part: {part_text}") for _ in range(ATTEMPTS)))
        attempts = await scored(len(graph), part_text, list(texts), [part.id], "generate")
        graph += attempts
        kept.append(max(attempts, key=lambda t: t.score).id)
    return {"graph": graph, "kept": kept}


async def aggregate(state: State) -> dict:
    # several thoughts in, one out: the new vertex has one parent per kept attempt
    material = "\n\n".join(f"Partial result {i}:\n{state['graph'][k].text}" for i, k in enumerate(state["kept"], 1))
    texts = await asyncio.gather(*(write(AGGREGATOR, f"Task: {state['task']}\n\n{material}") for _ in range(MERGES)))
    merges = await scored(len(state["graph"]), state["task"], list(texts), state["kept"], "aggregate")
    return {"graph": state["graph"] + merges, "kept": [max(merges, key=lambda t: t.score).id]}


async def refine(state: State) -> dict:
    best = state["graph"][state["kept"][0]]
    text = await write(REFINER, f"Task: {state['task']}\nDefects:\n" + "\n".join(best.defects) + f"\n\nResult:\n{best.text}")
    (new,) = await scored(len(state["graph"]), state["task"], [text], [best.id], "refine")
    keep = new if new.score > best.score else best  # refining can make it worse
    return {"graph": state["graph"] + [new], "kept": [keep.id], "refines": state["refines"] + 1}


def more(state: State) -> str:
    best = state["graph"][state["kept"][0]]
    return END if best.score >= GOOD_ENOUGH or state["refines"] >= MAX_REFINES else "refine"


builder = StateGraph(State)
builder.add_node("generate", generate)
builder.add_node("aggregate", aggregate)
builder.add_node("refine", refine)
builder.add_edge(START, "generate")
builder.add_edge("generate", "aggregate")
builder.add_conditional_edges("aggregate", more, ["refine", END])
builder.add_conditional_edges("refine", more, ["refine", END])
app = builder.compile()


async def run(task: str) -> dict:
    s = await app.ainvoke({"task": task, "graph": [], "kept": [], "refines": 0})
    best = s["graph"][s["kept"][0]]
    return {"answer": best.text, "score": best.score, "thoughts": [t.model_dump(exclude={"text"}) for t in s["graph"]]}

The wiring of the graph is the plan of operations: change it for your task, before the run. Write your own rubric, or score in code when the result can be checked.

Claude Agent SDK, python, package claude-agent-sdk
import asyncio

from claude_agent_sdk import ClaudeAgentOptions, ResultMessage, query
from pydantic import BaseModel

# the plan of operations, fixed before the run: split, attempt each part, merge, refine
MAX_PARTS, ATTEMPTS, MERGES, MAX_REFINES, GOOD_ENOUGH = 4, 3, 3, 2, 8
RUBRIC = "1. Covers everything in the material. 2. States each item once. 3. Adds nothing that is not in the material."
SPLITTER = f"Split the task into 2 to {MAX_PARTS} independent parts whose results can be merged."
GENERATOR = "Produce a complete result for your part only. Work from the material given and add nothing to it."
AGGREGATOR = ("Write a single result from the partial results. Keep everything they contain, state each item "
              "once and resolve contradictions explicitly. Add nothing that is in none of them.")
REFINER = "Rewrite the result so that each listed defect is fixed. Keep everything else. Add no new content."
SCORER = "Judge the result against the rubric. List concrete defects first, then score 1 to 10. Do not rewrite it."


class Parts(BaseModel):
    parts: list[str]


class Score(BaseModel):
    defects: list[str]
    score: int  # 1 to 10


class Thought(BaseModel):  # a vertex; parents are its incoming links
    id: int
    op: str
    text: str
    parents: list[int]
    score: int = 0
    defects: list[str] = []


async def ask(system: str, prompt: str, schema: type[BaseModel] | None = None) -> ResultMessage:
    # one query per call: a new session, so no agent sees another attempt or a score
    options = ClaudeAgentOptions(
        system_prompt=system,
        output_format={"type": "json_schema", "schema": schema.model_json_schema()} if schema else None)
    async for msg in query(prompt=prompt, options=options):
        if isinstance(msg, ResultMessage):
            return msg
    raise RuntimeError("no result")


async def write(system: str, prompt: str) -> str:
    return (await ask(system, prompt)).result or ""


async def run(task: str) -> dict:
    graph: list[Thought] = []  # every thought of the run, kept or dropped

    async def scored(goal: str, texts: list[str], parents: list[int], op: str) -> list[Thought]:
        # a fresh scorer per thought: the goal, the rubric and the text, nothing else
        raw = await asyncio.gather(*(ask(SCORER, f"Goal: {goal}\nRubric: {RUBRIC}\n\nResult:\n{t}", Score) for t in texts))
        new = []
        for text, msg in zip(texts, raw):
            s = Score.model_validate(msg.structured_output)
            new.append(Thought(id=len(graph) + len(new), op=op, text=text, parents=parents, score=s.score, defects=s.defects))
        graph.extend(new)
        return new

    # operation 1, generate: the parts, then several attempts per part; the best one survives
    split = Parts.model_validate((await ask(SPLITTER, task, Parts)).structured_output)
    kept: list[Thought] = []
    for part_text in split.parts[:MAX_PARTS]:
        part = Thought(id=len(graph), op="part", text=part_text, parents=[])
        graph.append(part)
        texts = await asyncio.gather(*(write(GENERATOR, f"Task: {task}\nYour part: {part_text}") for _ in range(ATTEMPTS)))
        kept.append(max(await scored(part_text, list(texts), [part.id], "generate"), key=lambda t: t.score))

    # operation 2, aggregate: several thoughts in, one out, with one parent per kept attempt
    material = "\n\n".join(f"Partial result {i}:\n{t.text}" for i, t in enumerate(kept, 1))
    texts = await asyncio.gather(*(write(AGGREGATOR, f"Task: {task}\n\n{material}") for _ in range(MERGES)))
    best = max(await scored(task, list(texts), [t.id for t in kept], "aggregate"), key=lambda t: t.score)

    # operation 3, refine: a loop on one thought, at most twice
    for _ in range(MAX_REFINES):
        if best.score >= GOOD_ENOUGH:
            break
        text = await write(REFINER, f"Task: {task}\nDefects:\n" + "\n".join(best.defects) + f"\n\nResult:\n{best.text}")
        (new,) = await scored(task, [text], [best.id], "refine")
        best = new if new.score > best.score else best  # refining can make it worse

    # the plan is exhausted: the kept thought is the result
    return {"answer": best.text, "score": best.score, "thoughts": [t.model_dump(exclude={"text"}) for t in graph]}

The body of run is the plan of operations: change it for your task, before the run. Write your own rubric, or score in code when the result can be checked.

OpenAI Agents SDK, python, package openai-agents
import asyncio

from agents import Agent, Runner
from pydantic import BaseModel

# the plan of operations, fixed before the run: split, attempt each part, merge, refine
MAX_PARTS, ATTEMPTS, MERGES, MAX_REFINES, GOOD_ENOUGH = 4, 3, 3, 2, 8
RUBRIC = "1. Covers everything in the material. 2. States each item once. 3. Adds nothing that is not in the material."


class Parts(BaseModel):
    parts: list[str]


class Score(BaseModel):
    defects: list[str]
    score: int  # 1 to 10


class Thought(BaseModel):  # a vertex; parents are its incoming links
    id: int
    op: str
    text: str
    parents: list[int]
    score: int = 0
    defects: list[str] = []


splitter = Agent(
    name="Splitter",
    instructions=f"Split the task into 2 to {MAX_PARTS} independent parts whose results can be merged.",
    output_type=Parts,
)
generator = Agent(
    name="Generator",
    instructions="Produce a complete result for your part only. Work from the material given and add nothing to it.",
)
aggregator = Agent(
    name="Aggregator",
    instructions="Write a single result from the partial results. Keep everything they contain, state each item "
    "once and resolve contradictions explicitly. Add nothing that is in none of them.",
)
refiner = Agent(
    name="Refiner",
    instructions="Rewrite the result so that each listed defect is fixed. Keep everything else. Add no new content.",
)
scorer = Agent(
    name="Scorer",
    instructions="Judge the result against the rubric. List concrete defects first, then score 1 to 10. Do not rewrite it.",
    output_type=Score,
)


async def write(agent: Agent, prompt: str) -> str:
    return str((await Runner.run(agent, prompt)).final_output)  # a fresh run: this prompt only


async def run(task: str) -> dict:
    graph: list[Thought] = []  # every thought of the run, kept or dropped

    async def scored(goal: str, texts: list[str], parents: list[int], op: str) -> list[Thought]:
        # a fresh scorer run per thought: the goal, the rubric and the text, nothing else
        runs = await asyncio.gather(*(
            Runner.run(scorer, f"Goal: {goal}\nRubric: {RUBRIC}\n\nResult:\n{t}") for t in texts))
        new = []
        for text, r in zip(texts, runs):
            s = r.final_output_as(Score)
            new.append(Thought(id=len(graph) + len(new), op=op, text=text, parents=parents, score=s.score, defects=s.defects))
        graph.extend(new)
        return new

    # operation 1, generate: the parts, then several attempts per part; the best one survives
    split = (await Runner.run(splitter, task)).final_output_as(Parts)
    kept: list[Thought] = []
    for part_text in split.parts[:MAX_PARTS]:
        part = Thought(id=len(graph), op="part", text=part_text, parents=[])
        graph.append(part)
        texts = await asyncio.gather(*(write(generator, f"Task: {task}\nYour part: {part_text}") for _ in range(ATTEMPTS)))
        kept.append(max(await scored(part_text, list(texts), [part.id], "generate"), key=lambda t: t.score))

    # operation 2, aggregate: several thoughts in, one out, with one parent per kept attempt
    material = "\n\n".join(f"Partial result {i}:\n{t.text}" for i, t in enumerate(kept, 1))
    texts = await asyncio.gather(*(write(aggregator, f"Task: {task}\n\n{material}") for _ in range(MERGES)))
    best = max(await scored(task, list(texts), [t.id for t in kept], "aggregate"), key=lambda t: t.score)

    # operation 3, refine: a loop on one thought, at most twice
    for _ in range(MAX_REFINES):
        if best.score >= GOOD_ENOUGH:
            break
        text = await write(refiner, f"Task: {task}\nDefects:\n" + "\n".join(best.defects) + f"\n\nResult:\n{best.text}")
        (new,) = await scored(task, [text], [best.id], "refine")
        best = new if new.score > best.score else best  # refining can make it worse

    # the plan is exhausted: the kept thought is the result
    return {"answer": best.text, "score": best.score, "thoughts": [t.model_dump(exclude={"text"}) for t in graph]}

The body of run is the plan of operations: change it for your task, before the run. Write your own rubric, or score in code when the result can be checked.

Vercel AI SDK, typescript, package ai
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';

const model = anthropic('claude-sonnet-5-5');
// the plan of operations, fixed before the run: split, attempt each part, merge, refine
const [MAX_PARTS, ATTEMPTS, MERGES, MAX_REFINES, GOOD_ENOUGH] = [4, 3, 3, 2, 8];
const RUBRIC = '1. Covers everything in the material. 2. States each item once. 3. Adds nothing that is not in the material.';
const GENERATOR = 'Produce a complete result for your part only. Work from the material given and add nothing to it.';
const AGGREGATOR =
  'Write a single result from the partial results. Keep everything they contain, state each item ' +
  'once and resolve contradictions explicitly. Add nothing that is in none of them.';
const REFINER = 'Rewrite the result so that each listed defect is fixed. Keep everything else. Add no new content.';
const SCORER = 'Judge the result against the rubric. List concrete defects first, then score 1 to 10. Do not rewrite it.';

const partsSchema = z.object({ parts: z.array(z.string()) });
const scoreSchema = z.object({ defects: z.array(z.string()), score: z.number().int().min(1).max(10) });
// a vertex; parents are its incoming links
type Thought = { id: number; op: string; text: string; parents: number[]; score: number; defects: string[] };
const best = (thoughts: Thought[]) => thoughts.reduce((a, b) => (b.score > a.score ? b : a));

// a fresh call each time: these instructions and this prompt only
async function write(instructions: string, prompt: string): Promise<string> {
  const { text } = await generateText({ model, instructions, prompt });
  return text;
}

export async function run(task: string) {
  const graph: Thought[] = []; // every thought of the run, kept or dropped

  async function scored(goal: string, texts: string[], parents: number[], op: string): Promise<Thought[]> {
    // a fresh scorer call per thought: the goal, the rubric and the text, nothing else
    const scores = await Promise.all(texts.map(async (text) => {
      const { output } = await generateText({
        model,
        instructions: SCORER,
        prompt: `Goal: ${goal}\nRubric: ${RUBRIC}\n\nResult:\n${text}`,
        output: Output.object({ schema: scoreSchema }),
      });
      return output;
    }));
    const fresh = texts.map((text, i) => ({ id: graph.length + i, op, text, parents, ...scores[i] }));
    graph.push(...fresh);
    return fresh;
  }

  // operation 1, generate: the parts, then several attempts per part; the best one survives
  const { output: split } = await generateText({
    model,
    instructions: `Split the task into 2 to ${MAX_PARTS} independent parts whose results can be merged.`,
    prompt: task,
    output: Output.object({ schema: partsSchema }),
  });
  const kept: Thought[] = [];
  for (const partText of split.parts.slice(0, MAX_PARTS)) {
    const part: Thought = { id: graph.length, op: 'part', text: partText, parents: [], score: 0, defects: [] };
    graph.push(part);
    const texts = await Promise.all(
      Array.from({ length: ATTEMPTS }, () => write(GENERATOR, `Task: ${task}\nYour part: ${partText}`)));
    kept.push(best(await scored(partText, texts, [part.id], 'generate')));
  }

  // operation 2, aggregate: several thoughts in, one out, with one parent per kept attempt
  const material = kept.map((t, i) => `Partial result ${i + 1}:\n${t.text}`).join('\n\n');
  const merges = await Promise.all(
    Array.from({ length: MERGES }, () => write(AGGREGATOR, `Task: ${task}\n\n${material}`)));
  let result = best(await scored(task, merges, kept.map((t) => t.id), 'aggregate'));

  // operation 3, refine: a loop on one thought, at most twice
  for (let i = 0; i < MAX_REFINES && result.score < GOOD_ENOUGH; i++) {
    const text = await write(REFINER, `Task: ${task}\nDefects:\n${result.defects.join('\n')}\n\nResult:\n${result.text}`);
    const [next] = await scored(task, [text], [result.id], 'refine');
    if (next.score > result.score) result = next; // refining can make it worse
  }

  // the plan is exhausted: the kept thought is the result
  return {
    answer: result.text,
    score: result.score,
    thoughts: graph.map(({ id, op, parents, score }) => ({ id, op, parents, score })),
  };
}

The body of run is the plan of operations: change it for your task, before the run. Write your own rubric, or score in code when the result can be checked.

Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.

What goes wrong

  • An aggregator given too many thoughts drops content silently. Merge a few at a time, and have the rubric check that nothing was lost.
  • Parts that overlap produce duplicates that the merge must then undo. Split so that each item belongs to one part.
  • A scorer that rates every attempt 7 or 8 makes keeping the best a random pick. Ask for defects before the score.
  • Refining can make a result worse. Score the new version and keep the old one when it rated higher.
  • The plan fixes the number of calls. Count them before the run. Do not let the model add operations.

Chain it

Its result can feed

Source