1What if the work needs many tool calls and bulky intermediate data?
Each turn the agent writes a short program instead of one tool call. A sandbox runs it. The program calls the tools as functions, loops over them and keeps intermediate data in variables. Only what it prints goes back to the agent, which then answers or writes the next program.
You may need it if
- The agent copies hundreds of rows from one tool result into the arguments of the next call.
- A job takes twenty tool rounds where a loop of five lines would do.
- Most of the context is raw tool output that the agent only needed to filter or count.
2When is writing a program the wrong way to call tools?
Use it when
- The task chains many calls, and each one depends on the result of the one before.
- Tool results are large and the agent only needs a count, a sample or a filtered part.
- The same operation is repeated across many items, which a loop states in a few lines.
- You can run a sandbox with no open network, time and memory limits, and a log of every run.
Avoid it when
- The job is one lookup or a few small calls with short results. A structured tool call is simpler and needs no sandbox.
Use Tool Use instead - The agent must read and weigh every intermediate result before choosing the next step. A program hides them from it.
Use ReAct instead - The calls change records, send messages or spend money, and each one should be reviewed before it runs.
Use Dry Run instead - You cannot isolate generated code, with resource limits and monitoring. Do not run it on a machine that holds credentials.
What it costs
- Model calls
- 1 + programs run (typically 1 to 4); tool calls made by a program cost no model call
- Latency
- low
- Complexity
- medium
3How does a program stand between the agent and the tools?
A loop around a sandbox. The agent writes a program or gives its final answer. A gate checks the program and the runs left. The sandbox executes it, the program calls the tool functions as often as it needs, and only its printed output or its error returns to the agent.
- Step 1 of 5
The agent writes a program
The task is "Which customers paid their last three invoices late, and how much do they still owe?". The agent sees two functions, list_customers and list_invoices. It writes fifteen lines that loop over the customers, keep the late payers and print a count and a total.
- Step 2 of 5
The program is checked
Before anything runs, the gate checks that the program calls only the two listed functions, that its loop is bounded and that runs remain. A rejected program goes back with a one-line reason.
- Step 3 of 5
The sandbox runs it
The program calls list_invoices once per customer, 300 calls in one run. The harness makes each real call. The 4,000 invoice rows land in variables inside the sandbox, and none of them enters the agent's context.
- Step 4 of 5
Only the printed output returns
The agent receives three lines, 12 customers, 8,460 euros in total, and the first three names. Had a call failed, the error text would come back instead and the agent would fix the program.
- Step 5 of 5
The agent answers, or writes again
With the printed figures the agent answers. If it needs the full list of names, a second program can print it, reusing the variables when the sandbox is kept between runs. After four programs the loop ends and the answer is marked incomplete.
What each part does
- Task
- The request or question, plus the signatures of the tool functions the agent may call from code.
- Agent
- Writes one program per turn against the tool functions, or gives the final answer once the printed output is enough.
- Run allowed?
- Checks the program calls only listed functions and that runs remain. A rejected program goes back with the reason.
- Code sandbox
- An isolated interpreter with time and memory limits and no open network. Runs the program and collects what it prints.
- Tool functions
- Your tools, exposed as functions inside the sandbox. The harness makes the real calls with your credentials.
- Answer
- The final reply, with the programs that ran and what each one printed.
- Agent → Run allowed?: writes a program
- Run allowed? → Code sandbox: allowed, runs left
- Run allowed? → Agent: rejected with the reason
- Code sandbox → Tool functions: function calls, many per run
- Tool functions → Code sandbox: results kept in variables
- Code sandbox → Agent: printed output or error
- Agent → Answer: final answer
- Run allowed? → Answer: run limit reached
Where it connects
A request or question that takes several tool calls or bulky data to settle. The tool functions, their signatures and the sandbox limits come with it.
When: The agent gives a final answer, or the run limit is reached.
The final reply, each program that ran with its printed output, and a flag saying whether the run limit cut the work short.
4What do I ask my assistant to set up?
The agents to launch
Code-writing agent
Solve the task by writing short Python programs that call the listed functions. Keep bulky data in variables and print only what you need to read: counts, totals, a few sample rows, errors. Never state a value you did not print. When the printed output settles the task, give your final answer instead of a program.
Sees: The task, the function signatures with an example each, and the printed output of earlier programs. Not the raw tool results or the credentials.
Program checker
Check one program before it runs. Reject it if it calls a function that is not listed, imports a network or file module, loops without a bound, or prints whole datasets. Give a one-line reason. Do not judge whether the task is solved.
Sees: The function list, the proposed program and the number of earlier runs. Not the agent's conversation.
A prompt you can paste
Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.
You are the orchestrator of a multi-agent workflow that follows the "Code as Action" architecture. Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it. # Task [Describe your task here] # Code as Action The agent writes a program that calls the tools and reports back. Input: the task above. It must provide: A request or question that takes several tool calls or bulky data to settle. The tool functions, their signatures and the sandbox limits come with it. If something essential is missing, ask for it before starting. Agents to launch: - Code-writing agent: Solve the task by writing short Python programs that call the listed functions. Keep bulky data in variables and print only what you need to read: counts, totals, a few sample rows, errors. Never state a value you did not print. When the printed output settles the task, give your final answer instead of a program. Context for this agent: The task, the function signatures with an example each, and the printed output of earlier programs. Not the raw tool results or the credentials. - Program checker: Check one program before it runs. Reject it if it calls a function that is not listed, imports a network or file module, loops without a bound, or prints whole datasets. Give a one-line reason. Do not judge whether the task is solved. Context for this agent: The function list, the proposed program and the number of earlier runs. Not the agent's conversation. Procedure: 1. Expose each tool as a function inside the sandbox, with a typed signature, a one-line description and an example. Make the real calls from the harness, so that credentials never enter the sandbox. 2. Set the sandbox limits. No open network, 30 seconds and a memory cap per run, and at most 2,000 characters of printed output returned to the agent. 3. Launch the code-writing agent on the task with the function signatures. 4. For each program, launch the program checker. If it rejects, return the reason to the agent. If it allows, run the program in the sandbox. 5. Return only the printed output, or the error message, to the agent. Keep the same sandbox for the whole run so that later programs can reuse variables. 6. Count runs. After 4 programs, stop running code and ask the agent for its best final answer with what it has. Stop rule: Stop when the agent gives a final answer, or after 4 programs have run, whichever comes first. At the limit, ask once for a final answer and mark it incomplete. Exit: - `answer`, when: The agent gives a final answer, or the run limit is reached. Deliver: The final reply, each program that ran with its printed output, and a flag saying whether the run limit cut the work short. # Final deliverable The final answer, then each program that ran with its printed output, and whether the run limit was reached.
The same thing in code
import operator
from typing import Annotated, TypedDict
from langchain.chat_models import init_chat_model
from langchain_core.messages import AIMessage, AnyMessage, HumanMessage, SystemMessage, ToolCall, ToolMessage
from langchain_core.tools import tool
from langgraph.graph import END, START, StateGraph
from langgraph.graph.message import add_messages
from pydantic import BaseModel
MAX_RUNS = 4
MAX_OUTPUT = 2000 # characters of printed output returned to the agent
FUNCTIONS = ('list_customers() -> list[dict]: every customer, with "id" and "name"\n'
'list_invoices(customer_id: str) -> list[dict]: invoices with "amount", "due" and "paid_on"')
AGENT = ("Solve the task by writing short Python programs that call these functions:\n" + FUNCTIONS +
"\nKeep bulky data in variables and print only what you need to read: counts, totals, a few "
"sample rows, errors. Never state a value you did not print. When the printed output settles "
"the task, give your final answer instead of a program.")
CHECKER = ("Check one program before it runs. Reject it if it calls a function that is not listed, imports "
"a network or file module, loops without a bound, or prints whole datasets. Give a one-line reason.")
async def run_in_sandbox(code: str) -> str:
# plug in your sandbox: no open network, time and memory limits, FUNCTIONS defined inside and
# served by the harness; keep one interpreter per run so that variables survive between programs
return "stub: what the program printed"
@tool
def run_program(code: str) -> str:
"""Run a Python program in the sandbox and return what it prints."""
return "" # schema only: the graph checks the program, then runs it in the sandbox
class Verdict(BaseModel):
allowed: bool
reason: str
class State(TypedDict):
messages: Annotated[list[AnyMessage], add_messages]
runs: Annotated[list[dict], operator.add]
approved: list[ToolCall]
limit_hit: bool
model = init_chat_model("anthropic:claude-sonnet-5-5")
agent_model = model.bind_tools([run_program])
checker_model = model.with_structured_output(Verdict)
def last_ai(state: State) -> AIMessage:
msg = state["messages"][-1]
assert isinstance(msg, AIMessage)
return msg
async def agent(state: State) -> dict:
return {"messages": [await agent_model.ainvoke([SystemMessage(AGENT), *state["messages"]])]}
async def check(state: State) -> dict:
calls = last_ai(state).tool_calls
if len(state["runs"]) + len(calls) > MAX_RUNS: # the gate: at most 4 programs
return {"limit_hit": True, "approved": []}
approved, rejected = [], []
for call in calls:
# the checker sees the function list and the program, not the agent's conversation
prompt = f"Functions:\n{FUNCTIONS}\n\nProgram:\n{call['args'].get('code', '')}\n\nEarlier runs: {len(state['runs'])}"
verdict = Verdict.model_validate(await checker_model.ainvoke([SystemMessage(CHECKER), HumanMessage(prompt)]))
if verdict.allowed:
approved.append(call)
else:
rejected.append(ToolMessage(f"Rejected: {verdict.reason}", tool_call_id=call["id"] or ""))
return {"approved": approved, "messages": rejected}
async def sandbox(state: State) -> dict:
results, runs = [], []
for call in state["approved"]:
code = str(call["args"].get("code", ""))
printed = (await run_in_sandbox(code))[:MAX_OUTPUT] # only printed output goes back
results.append(ToolMessage(printed, tool_call_id=call["id"] or ""))
runs.append({"program": code, "printed": printed})
return {"messages": results, "runs": runs}
async def final(state: State) -> dict:
skipped = [ToolMessage("Not run: run limit reached.", tool_call_id=c["id"] or "") for c in last_ai(state).tool_calls]
nudge = HumanMessage("Run limit reached. Give your best final answer now with what you have.")
reply = await agent_model.ainvoke([SystemMessage(AGENT), *state["messages"], *skipped, nudge])
return {"messages": [*skipped, nudge, reply]}
def after_check(state: State) -> str:
return "final" if state["limit_hit"] else "sandbox" if state["approved"] else "agent"
builder = StateGraph(State)
for name, node in {"agent": agent, "check": check, "sandbox": sandbox, "final": final}.items():
builder.add_node(name, node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", lambda s: "check" if last_ai(s).tool_calls else END, ["check", END])
builder.add_conditional_edges("check", after_check, ["sandbox", "agent", "final"])
builder.add_edge("sandbox", "agent")
builder.add_edge("final", END)
graph = builder.compile()
async def run(task: str) -> dict:
state = await graph.ainvoke(
{"messages": [HumanMessage(task)], "runs": [], "approved": [], "limit_hit": False},
{"recursion_limit": 40},
)
return {"answer": state["messages"][-1].text, "runs": state["runs"], "incomplete": state["limit_hit"]}
run_in_sandbox is a stub. Plug in an isolated interpreter that defines the tool functions and routes their calls back to your harness. Never use exec in your own process.
from typing import Any
from claude_agent_sdk import (
AgentDefinition, ClaudeAgentOptions, PermissionResultAllow, PermissionResultDeny,
ResultMessage, ToolPermissionContext, create_sdk_mcp_server, query, tool,
)
from pydantic import BaseModel
MAX_RUNS = 4
MAX_OUTPUT = 2000 # characters of printed output returned to the agent
RUN_TOOL = "mcp__sandbox__run_program"
FUNCTIONS = ('list_customers() -> list[dict]: every customer, with "id" and "name"\n'
'list_invoices(customer_id: str) -> list[dict]: invoices with "amount", "due" and "paid_on"')
async def run_in_sandbox(code: str) -> str:
# plug in your sandbox: no open network, time and memory limits, FUNCTIONS defined inside and
# served by the harness; keep one interpreter per run so that variables survive between programs
return "stub: what the program printed"
AGENTS = {
"coder": AgentDefinition(
description="Solves the task by writing programs for the sandbox.",
prompt="Solve the task by writing short Python programs and running them with run_program. They "
"can call these functions:\n" + FUNCTIONS + "\nKeep bulky data in variables and print only "
"what you need to read: counts, totals, a few sample rows, errors. Never state a value "
"you did not print. When the printed output settles the task, give your final answer.",
tools=[RUN_TOOL],
),
"program-checker": AgentDefinition(
description="Allows or rejects one program before it runs.",
prompt="Check one program before it runs. Reject it if it calls a function that is not listed, "
"imports a network or file module, loops without a bound, or prints whole datasets. "
"Give a one-line reason.",
),
}
class Verdict(BaseModel):
allowed: bool
reason: str
async def result_of(prompt: str, options: ClaudeAgentOptions) -> ResultMessage:
async for msg in query(prompt=prompt, options=options):
if isinstance(msg, ResultMessage):
return msg
raise RuntimeError("no result")
async def run(task: str) -> dict:
runs: list[dict] = []
limit_hit = False
@tool("run_program", "Run a Python program in the sandbox and return what it prints.", {"code": str})
async def run_program(args: dict[str, Any]) -> dict[str, Any]:
printed = (await run_in_sandbox(args["code"]))[:MAX_OUTPUT] # only printed output goes back
runs.append({"program": args["code"], "printed": printed})
return {"content": [{"type": "text", "text": printed}]}
async def gate(name: str, args: dict[str, Any], ctx: ToolPermissionContext) -> PermissionResultAllow | PermissionResultDeny:
nonlocal limit_hit
if name != RUN_TOOL:
return PermissionResultDeny(message="Only run_program is available.")
if len(runs) >= MAX_RUNS: # the gate: at most 4 programs
limit_hit = True
return PermissionResultDeny(message="Run limit reached: give your best final answer now.")
# the checker runs in its own session: function list and program only
check = await result_of(
"Use the program-checker agent on this program and return its verdict.\n\n"
f"Functions:\n{FUNCTIONS}\n\nProgram:\n{args.get('code', '')}\n\nEarlier runs: {len(runs)}",
ClaudeAgentOptions(agents=AGENTS, allowed_tools=["Agent"],
output_format={"type": "json_schema", "schema": Verdict.model_json_schema()}),
)
verdict = Verdict.model_validate(check.structured_output)
if not verdict.allowed:
return PermissionResultDeny(message=f"Rejected: {verdict.reason}")
return PermissionResultAllow()
options = ClaudeAgentOptions(
agents=AGENTS,
mcp_servers={"sandbox": create_sdk_mcp_server("sandbox", tools=[run_program])},
allowed_tools=["Agent"], # run_program is not pre-approved, so each program reaches the gate
can_use_tool=gate,
)
answer = await result_of(f"Use the coder agent on this task and return its answer:\n\n{task}", options)
return {"answer": answer.result, "runs": runs, "incomplete": limit_hit}
run_in_sandbox is a stub. Plug in an isolated interpreter that defines the tool functions and routes their calls back to your harness. Keep the SDK's own shell and file tools away from the coder.
from dataclasses import dataclass, field
from agents import (
Agent, RunContextWrapper, Runner, ToolGuardrailFunctionOutput, ToolInputGuardrailData,
function_tool, tool_input_guardrail,
)
from pydantic import BaseModel
MAX_RUNS = 4
MAX_OUTPUT = 2000 # characters of printed output returned to the agent
FUNCTIONS = ('list_customers() -> list[dict]: every customer, with "id" and "name"\n'
'list_invoices(customer_id: str) -> list[dict]: invoices with "amount", "due" and "paid_on"')
async def run_in_sandbox(code: str) -> str:
# plug in your sandbox: no open network, time and memory limits, FUNCTIONS defined inside and
# served by the harness; keep one interpreter per run so that variables survive between programs
return "stub: what the program printed"
class Verdict(BaseModel):
allowed: bool
reason: str
@dataclass
class RunLog:
runs: list[dict] = field(default_factory=list)
limit_hit: bool = False
checker = Agent(
name="Program checker",
instructions="Check one program before it runs. Reject it if it calls a function that is not listed, "
"imports a network or file module, loops without a bound, or prints whole datasets. "
"Give a one-line reason.",
output_type=Verdict,
)
@tool_input_guardrail
async def check_program(data: ToolInputGuardrailData) -> ToolGuardrailFunctionOutput:
log: RunLog = data.context.context
if len(log.runs) >= MAX_RUNS: # the gate: at most 4 programs
log.limit_hit = True
return ToolGuardrailFunctionOutput.reject_content("Run limit reached: give your best final answer now.")
# the checker sees the function list and the program, not the agent's conversation
prompt = f"Functions:\n{FUNCTIONS}\n\nProgram (as tool arguments):\n{data.context.tool_arguments}\n\nEarlier runs: {len(log.runs)}"
verdict = (await Runner.run(checker, prompt)).final_output_as(Verdict)
if not verdict.allowed:
return ToolGuardrailFunctionOutput.reject_content(f"Rejected: {verdict.reason}")
return ToolGuardrailFunctionOutput.allow()
@function_tool(tool_input_guardrails=[check_program])
async def run_program(ctx: RunContextWrapper[RunLog], code: str) -> str:
"""Run a Python program in the sandbox and return what it prints."""
printed = (await run_in_sandbox(code))[:MAX_OUTPUT] # only printed output goes back
ctx.context.runs.append({"program": code, "printed": printed})
return printed
coder = Agent[RunLog](
name="Code-writing agent",
instructions="Solve the task by writing short Python programs and running them with run_program. They "
"can call these functions:\n" + FUNCTIONS + "\nKeep bulky data in variables and print "
"only what you need to read: counts, totals, a few sample rows, errors. Never state a "
"value you did not print. When the printed output settles the task, give your final answer.",
tools=[run_program],
)
async def run(task: str) -> dict:
log = RunLog()
result = await Runner.run(coder, task, context=log, max_turns=MAX_RUNS + 6)
return {"answer": result.final_output, "runs": log.runs, "incomplete": log.limit_hit}
run_in_sandbox is a stub. Plug in an isolated interpreter that defines the tool functions and routes their calls back to your harness. Never use exec in your own process.
import { anthropic } from '@ai-sdk/anthropic';
import { isStepCount, Output, tool, ToolLoopAgent } from 'ai';
import { z } from 'zod';
const MAX_RUNS = 4;
const MAX_OUTPUT = 2000; // characters of printed output returned to the agent
const FUNCTIONS =
'listCustomers(): Promise<{ id: string; name: string }[]>\n' +
'listInvoices(customerId: string): Promise<{ amount: number; due: string; paidOn: string | null }[]>';
const model = anthropic('claude-sonnet-5-5');
// stub: plug in your sandbox. No open network, time and memory limits, FUNCTIONS defined inside
// and served by the harness; keep one interpreter per run so that variables survive between programs.
const runInSandbox = async (code: string) => `stub: what the program printed (${code.length} characters run)`;
const checker = new ToolLoopAgent({
model,
instructions:
'Check one program before it runs. Reject it if it calls a function that is not listed, imports ' +
'a network or file module, loops without a bound, or prints whole datasets. Give a one-line reason.',
output: Output.object({ schema: z.object({ allowed: z.boolean(), reason: z.string() }) }),
});
export async function run(task: string) {
const runs: { program: string; printed: string }[] = [];
let limitHit = false;
// the gate: every program passes the checker, and at most 4 programs run
async function gated(code: string) {
if (runs.length >= MAX_RUNS) {
limitHit = true;
return 'Not run: run limit reached. Give your best final answer now.';
}
const { output } = await checker.generate({
prompt: `Functions:\n${FUNCTIONS}\n\nProgram:\n${code}\n\nEarlier runs: ${runs.length}`,
});
if (!output.allowed) return `Rejected: ${output.reason}`;
const printed = (await runInSandbox(code)).slice(0, MAX_OUTPUT); // only printed output goes back
runs.push({ program: code, printed });
return printed;
}
const agent = new ToolLoopAgent({
model,
instructions:
'Solve the task by writing short TypeScript programs and running them with runProgram. They can ' +
`call these functions:\n${FUNCTIONS}\nKeep bulky data in variables and print only what you need ` +
'to read: counts, totals, a few sample rows, errors. Never state a value you did not print. ' +
'When the printed output settles the task, give your final answer.',
tools: {
runProgram: tool({
description: 'Run a TypeScript program in the sandbox and return what it prints with console.log.',
inputSchema: z.object({ code: z.string() }),
execute: (input) => gated(input.code),
}),
},
prepareStep: async () => (limitHit ? { toolChoice: 'none' as const } : {}), // at the limit, answer only
stopWhen: isStepCount(MAX_RUNS + 6),
});
const { text } = await agent.generate({ prompt: task });
return { answer: text, runs, incomplete: limitHit };
}
runInSandbox is a stub. Plug in an isolated runtime that defines the tool functions and routes their calls back to your harness. Never use eval in your own process.
Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.
What goes wrong
- Generated code can do whatever the sandbox allows. Run it isolated, with no open network and no credentials inside, and log every run.
- The agent may state a value it never printed, as if it had read the variable. Require every figure in the answer to appear in a printed output.
- A program that prints a whole table brings the bulk back into the context. Cap the output returned and say how much was cut.
- Some sandboxes start empty for each program, so variables do not survive. Then have the program save what it needs to a file, or fetch it again.
- A write function called in a loop repeats one mistake across every item. Expose read functions only, or put the writes behind their own gate.
Chain it
Its result can feed
- Reflection conditional
Condition: Use the link only when the task asks for a written answer such as a report, summary or explanation, not for actions on the tools or a single computed value.
- Chain of Verification conditional
Condition: Use the link only when A's final reply answers a question with factual claims, not when it reports actions the programs carried out.
- Debate conditional
Condition: Use the link only when the task is a question or decision to settle, not a request to perform actions through the tools.
- Constitutional Critique conditional
Condition: Use the link only when A's final reply is a text answer to a question, not the outcome of actions the programs executed.
- Textual Backpropagation conditional
Condition: Use the link only when the task asks for a written answer such as a report or explanation, not for actions on the tools or a single computed value.