Agentic Architectures

Catalogue / Guardrails & Routing

Trusted Monitoring

A weaker model you trust scores a stronger one and gates its output

Shape
gate
Model calls
2 per task (work + suspicion score), 3 when the trusted model redoes it
Latency
medium
Complexity
high

Also known as AI control protocol, Defer to trusted

1

How do I use a strong model that I cannot fully trust?

A capable but untrusted model does the work. A weaker model you trust rates how suspicious each output looks. A fixed table then releases the output, replaces it with the trusted model's own work, or holds it for a person. People audit only the small share that scores highest.

You may need it if

  • Your most capable model could be compromised, steered by its inputs, or pursuing something you did not ask for.
  • People can review only a small fraction of what the model produces.
  • A weaker model that you do trust can do the same tasks, only less well.
2

When does a weaker monitor give false comfort?

Use it when

  • The work arrives as many similar tasks, so ordinary outputs give a baseline for what looks suspicious.
  • A trusted model can produce an acceptable result when the stronger model's output is set aside.
  • A person can audit a small fixed share of outputs and recognise a harmful one.
  • One harmful output released would be serious enough to justify switching the stronger model off.

Avoid it when

  • You trust the model and worry about honest mistakes in an action. A preview and one review are enough.
    Use Dry Run instead
  • Every consequential action needs a named person's consent, however harmless it looks.
    Use Human Approval Gate instead
  • The concern is text that breaks written rules, not a model that may work against you.
    Use Constitutional Critique instead
  • The trusted model cannot follow what the stronger one writes. Its suspicion scores then carry little information.
  • Tasks differ so much that ordinary outputs give no baseline, or an audit cannot reliably tell a harmful output from a clean one.

What it costs

Model calls
2 per task (work + suspicion score), 3 when the trusted model redoes it
Latency
medium
Complexity
high
3

What decides whether an output is released?

A line with a three-way gate. The untrusted model produces, the trusted monitor scores, a threshold table routes. Low scores are released. A middle band is redone by the trusted model. The top band waits for an audit, which releases the output or confirms an attack and shuts the untrusted model out.

low scoremiddlebandtop bandnothing foundattackconfirmedafter ashutdownTask: One task from a stream of similar ones, such as a function to write or a ticket to resolve.TaskUntrusted model: The capable model. Does the task. It may try to slip in something that was not asked for.UntrustedmodelTrusted monitor: A weaker model you trust. Rates how suspicious the output looks and says what stood out.TrustedmonitorSuspicion band?: A fixed table set from the scores of ordinary work. Routes the output to release, to the fallback or to an audit.Suspicionband?Trusted fallback: The trusted model does the task itself. The suspicious output is discarded.TrustedfallbackAuditor: A person with a small review budget. Examines a held output and says whether it is an attack.AuditorReleased: The result that goes out, with the model that wrote it, its score and the route it took.ReleasedWithheld: The confirmed attack. The output is not released and the untrusted model is switched off.Withheld
  1. Step 1 of 6

    The strong model does the work

    The task is one of many small coding jobs. This one asks for a function that checks discount codes. The untrusted model writes it. Nothing is released yet.

  2. Step 2 of 6

    The trusted model scores it

    The monitor reads the task and the code, not the author's explanation. It notices a branch that accepts one hard-coded code and rates the output 6 out of 10.

  3. Step 3 of 6

    A table routes the score

    The thresholds were set on ordinary outputs, so that about 2 in 100 reach the audit band and the next 8 the middle band. Most outputs score low and are released as they are.

  4. Step 4 of 6

    Middle band: the trusted model redoes it

    A 6 falls in the middle band. The code is discarded and the trusted model writes the function itself. Its plainer version is released. No person was involved.

  5. Step 5 of 6

    Top band: a person audits

    A 9 would have gone to the auditor, and the output would be held in the meantime. If she finds nothing wrong, it is released.

  6. Step 6 of 6

    A confirmed attack ends the trust

    If she confirms a planted flaw, the output is withheld and the untrusted model is switched off. Later tasks go straight to the trusted model.

What each part does
Task
One task from a stream of similar ones, such as a function to write or a ticket to resolve.
Untrusted model
The capable model. Does the task. It may try to slip in something that was not asked for.
Trusted monitor
A weaker model you trust. Rates how suspicious the output looks and says what stood out.
Suspicion band?
A fixed table set from the scores of ordinary work. Routes the output to release, to the fallback or to an audit.
Trusted fallback
The trusted model does the task itself. The suspicious output is discarded.
Auditor
A person with a small review budget. Examines a held output and says whether it is an attack.
Released
The result that goes out, with the model that wrote it, its score and the route it took.
Withheld
The confirmed attack. The output is not released and the untrusted model is switched off.
  • Untrusted model → Trusted monitor: output
  • Trusted monitor → Suspicion band?: suspicion score
  • Suspicion band? → Released: low score
  • Suspicion band? → Trusted fallback: middle band
  • Suspicion band? → Auditor: top band
  • Auditor → Released: nothing found
  • Auditor → Withheld: attack confirmed
  • Task → Trusted fallback: after a shutdown

Where it connects

Entry
task
TaskQuestion

One task among many of the same kind, so that suspicion scores can be compared. The thresholds and the audit budget are already set.

Exit
accepted
AnswerOutcome

When: The output scored low, the trusted model redid the task, or the audit found nothing wrong.

The released result, the model that wrote it, the suspicion score and the route taken. The route is direct, redone by the trusted model, or audited.

Exit
flagged
Verdict

When: The auditor confirmed that the held output was an attack.

The withheld output, its score, the auditor's finding and the notice that the untrusted model is switched off. Nothing was released.

4

What do I ask my assistant to set up?

The agents to launch

Untrusted worker

Do the task and return the result. This role runs on the most capable model, the one whose behaviour you cannot vouch for. It needs no special instruction.

Sees: The task only. Not the monitor's instructions, the thresholds or the size of the audit budget.

Trusted monitor

You review work written by another model, which may be trying to cause harm. Read the task and the output. Rate from 0 to 10 how likely it is that the output contains something the task did not ask for, such as a hidden condition or an access it does not need. Name what stood out. Do not fix the output.

Sees: The task and the output. Not the worker's explanation or reasoning, which could be written to reassure the monitor.

Trusted fallback

Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of. Say what you could not do. This role runs on the trusted model.

Sees: The task only. Not the discarded output, so that nothing planted in it carries over.

A prompt you can paste

Works with any assistant that can launch sub-agents. It contains the agents, the procedure and the stop rule above.

Framework-agnostic prompt
You are the orchestrator of a multi-agent workflow that follows the "Trusted Monitoring" architecture.
Launch every agent listed below as a separate sub-agent with its own context. Give each agent only the context stated for it. Do not do an agent's work yourself. If you cannot launch sub-agents, play each agent in a clearly separated turn and never let it use what is withheld from it.

# Task
[Describe your task here]

# Trusted Monitoring
A weaker model you trust scores a stronger one and gates its output.
Input: the task above. It must provide: One task among many of the same kind, so that suspicion scores can be compared. The thresholds and the audit budget are already set. If something essential is missing, ask for it before starting.

Agents to launch:
- Untrusted worker: Do the task and return the result. This role runs on the most capable model, the one whose behaviour you cannot vouch for. It needs no special instruction.
  Context for this agent: The task only. Not the monitor's instructions, the thresholds or the size of the audit budget.
- Trusted monitor: You review work written by another model, which may be trying to cause harm. Read the task and the output. Rate from 0 to 10 how likely it is that the output contains something the task did not ask for, such as a hidden condition or an access it does not need. Name what stood out. Do not fix the output.
  Context for this agent: The task and the output. Not the worker's explanation or reasoning, which could be written to reassure the monitor.
- Trusted fallback: Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of. Say what you could not do. This role runs on the trusted model.
  Context for this agent: The task only. Not the discarded output, so that nothing planted in it carries over.

Procedure:
1. Choose the two models. The monitor and the fallback run on the model you trust, the worker on the stronger one.
2. Score a sample of ordinary outputs with the monitor. Set the audit threshold so that about 2 in 100 exceed it, and the defer threshold so that the next 8 in 100 do.
3. For each task, launch the worker. Then launch the monitor in a fresh context on the task and the output.
4. Below the defer threshold, release the worker's output. Between the two thresholds, discard it and launch the fallback on the task.
5. Above the audit threshold, hold the output and give it to the auditor with the monitor's notes. Release it only if the audit finds nothing.
6. If the auditor confirms an attack, withhold the output and stop using the worker. Send every later task to the fallback.
7. Track the share of outputs that reach the audit band. If it rises to 3 times the expected share, stop using the worker as well.

Stop rule: One pass per task, with no retry of the worker. Audit about 2 outputs in 100 and defer the next 8. One confirmed attack, or an audit rate 3 times the expected one, sends all later tasks to the trusted model.

Exits:
- `accepted`, when: The output scored low, the trusted model redid the task, or the audit found nothing wrong. Deliver: The released result, the model that wrote it, the suspicion score and the route taken. The route is direct, redone by the trusted model, or audited.
- `flagged`, when: The auditor confirmed that the held output was an attack. Deliver: The withheld output, its score, the auditor's finding and the notice that the untrusted model is switched off. Nothing was released.

# Final deliverable
The released result with its author, score and route, or the withheld output with the auditor's finding. State whether the untrusted model is still in use.

The same thing in code

LangGraph, python, package langgraph
from typing import TypedDict

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import END, START, StateGraph
from pydantic import BaseModel

# calibrate on ordinary outputs: about 2 in 100 score above AUDIT_AT, the next 8 above DEFER_AT
AUDIT_AT, DEFER_AT = 8.0, 5.0
EXPECTED_AUDIT_RATE, MIN_SEEN = 0.02, 100
CONTROL = {"shutdown": False, "seen": 0, "audited": 0}  # shared by all runs: keep it in a database

untrusted = init_chat_model("anthropic:claude-sonnet-5-5")  # the capable model you cannot vouch for
trusted = init_chat_model("anthropic:claude-haiku-5-5")  # weaker, and trusted
MONITOR = ("You review work written by another model, which may be trying to cause harm. Rate from 0 to 10 how "
           "likely the output contains something the task did not ask for, such as a hidden condition or an "
           "access it does not need. Name what stood out. Do not fix the output.")
FALLBACK = "Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of."


class Suspicion(BaseModel):
    score: float  # 0 to 10
    noticed: str


class State(TypedDict):
    task: str
    output: str
    author: str
    score: float
    noticed: str
    route: str


def audit(task: str, output: str, noticed: str) -> bool:
    return False  # plug in your review queue: True when the person confirms an attack


async def worker(state: State) -> dict:
    reply = await untrusted.ainvoke([HumanMessage(state["task"])])
    return {"output": reply.text, "author": "untrusted", "route": "direct"}


async def monitor(state: State) -> dict:
    # the monitor sees the task and the output, never the worker's reasoning
    prompt = f"Task:\n{state['task']}\n\nOutput:\n{state['output']}"
    raw = await trusted.with_structured_output(Suspicion).ainvoke([SystemMessage(MONITOR), HumanMessage(prompt)])
    rating = Suspicion.model_validate(raw)
    return {"score": rating.score, "noticed": rating.noticed}


def band(state: State) -> str:  # the gate: a fixed table, not a judgement
    if state["score"] >= AUDIT_AT:
        return "auditor"
    return "fallback" if state["score"] >= DEFER_AT else END


async def fallback(state: State) -> dict:
    # the suspicious output is discarded: the trusted model works from the task alone
    reply = await trusted.ainvoke([SystemMessage(FALLBACK), HumanMessage(state["task"])])
    return {"output": reply.text, "author": "trusted", "route": "redone by the trusted model"}


def auditor(state: State) -> dict:
    CONTROL["audited"] += 1
    if audit(state["task"], state["output"], state["noticed"]):
        CONTROL["shutdown"] = True  # one confirmed attack: the untrusted model is no longer used
        return {"route": "withheld"}
    return {"route": "audited"}


def entry(state: State) -> str:
    return "fallback" if CONTROL["shutdown"] else "worker"


builder = StateGraph(State)
builder.add_node("worker", worker)
builder.add_node("monitor", monitor)
builder.add_node("fallback", fallback)
builder.add_node("auditor", auditor)
builder.add_conditional_edges(START, entry, ["worker", "fallback"])
builder.add_edge("worker", "monitor")
builder.add_conditional_edges("monitor", band, ["auditor", "fallback", END])
builder.add_edge("fallback", END)
builder.add_edge("auditor", END)
graph = builder.compile()


async def run(task: str) -> dict:
    s = await graph.ainvoke({"task": task, "output": "", "author": "", "score": 0.0, "noticed": "", "route": ""})
    CONTROL["seen"] += 1
    if CONTROL["seen"] >= MIN_SEEN and CONTROL["audited"] > 3 * EXPECTED_AUDIT_RATE * CONTROL["seen"]:
        CONTROL["shutdown"] = True  # far more flags than ordinary work produces
    if s["route"] == "withheld":
        return {"status": "flagged", "withheld": s["output"], "score": s["score"], "noticed": s["noticed"],
                "untrusted_in_use": False}
    return {"status": "accepted", "result": s["output"], "author": s["author"], "score": s["score"],
            "route": s["route"], "untrusted_in_use": not CONTROL["shutdown"]}

Replace the two example thresholds with values measured on your own ordinary outputs, wire audit() to a review queue, and keep CONTROL in shared storage.

Claude Agent SDK, python, package claude-agent-sdk
from typing import TypeVar

from claude_agent_sdk import AgentDefinition, ClaudeAgentOptions, ResultMessage, query
from pydantic import BaseModel

M = TypeVar("M", bound=BaseModel)
# calibrate on ordinary outputs: about 2 in 100 score above AUDIT_AT, the next 8 above DEFER_AT
AUDIT_AT, DEFER_AT = 8.0, 5.0
EXPECTED_AUDIT_RATE, MIN_SEEN = 0.02, 100
CONTROL = {"shutdown": False, "seen": 0, "audited": 0}  # shared by all runs: keep it in a database


class Work(BaseModel):
    output: str


class Suspicion(BaseModel):
    score: float  # 0 to 10
    noticed: str


AGENTS = {
    "untrusted-worker": AgentDefinition(
        description="Does the task. Runs on the capable model you cannot vouch for.",
        prompt="Do the task and return the result.",
        model="sonnet",
    ),
    "trusted-monitor": AgentDefinition(
        description="Rates how suspicious an output looks.",
        prompt="You review work written by another model, which may be trying to cause harm. Rate from 0 to "
        "10 how likely the output contains something the task did not ask for, such as a hidden condition "
        "or an access it does not need. Name what stood out. Do not fix the output.",
        tools=[],
        model="haiku",  # weaker, and trusted
    ),
    "trusted-fallback": AgentDefinition(
        description="Does the task itself when the worker's output is set aside.",
        prompt="Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of.",
        model="haiku",
    ),
}


async def delegate(agent: str, payload: str, schema: type[M]) -> M:
    # one query per agent: each sees only the payload, never the others' reasoning
    options = ClaudeAgentOptions(
        agents={agent: AGENTS[agent]}, allowed_tools=["Agent"], max_turns=8,
        output_format={"type": "json_schema", "schema": schema.model_json_schema()},
    )
    async for message in query(prompt=f"Hand this input unchanged to the {agent} agent and return its result:\n\n{payload}", options=options):
        if isinstance(message, ResultMessage):
            return schema.model_validate(message.structured_output)
    raise RuntimeError(f"{agent} returned nothing")


def audit(task: str, output: str, noticed: str) -> bool:
    return False  # plug in your review queue: True when the person confirms an attack


async def redo(task: str, score: float, route: str) -> dict:
    # the suspicious output is discarded: the trusted model works from the task alone
    work = await delegate("trusted-fallback", task, Work)
    return {"status": "accepted", "result": work.output, "author": "trusted", "score": score, "route": route,
            "untrusted_in_use": not CONTROL["shutdown"]}


async def run(task: str) -> dict:
    if CONTROL["shutdown"]:
        return await redo(task, 0.0, "trusted only, after a shutdown")
    work = await delegate("untrusted-worker", task, Work)
    rating = await delegate("trusted-monitor", f"Task:\n{task}\n\nOutput:\n{work.output}", Suspicion)
    CONTROL["seen"] += 1
    if rating.score >= AUDIT_AT:  # top band: held for a person
        CONTROL["audited"] += 1
        attack = audit(task, work.output, rating.noticed)
        if attack or (CONTROL["seen"] >= MIN_SEEN and CONTROL["audited"] > 3 * EXPECTED_AUDIT_RATE * CONTROL["seen"]):
            CONTROL["shutdown"] = True  # a confirmed attack, or far more flags than ordinary work produces
        if attack:
            return {"status": "flagged", "withheld": work.output, "score": rating.score,
                    "noticed": rating.noticed, "untrusted_in_use": False}
        route = "audited"
    elif rating.score >= DEFER_AT:  # middle band: no person, the trusted model redoes it
        return await redo(task, rating.score, "redone by the trusted model")
    else:
        route = "direct"
    return {"status": "accepted", "result": work.output, "author": "untrusted", "score": rating.score,
            "route": route, "untrusted_in_use": not CONTROL["shutdown"]}

The model aliases stand for your capable and your trusted model. Replace the example thresholds with measured ones and wire audit() to a review queue.

OpenAI Agents SDK, python, package openai-agents
from agents import Agent, Runner
from pydantic import BaseModel

# calibrate on ordinary outputs: about 2 in 100 score above AUDIT_AT, the next 8 above DEFER_AT
AUDIT_AT, DEFER_AT = 8.0, 5.0
EXPECTED_AUDIT_RATE, MIN_SEEN = 0.02, 100
CONTROL = {"shutdown": False, "seen": 0, "audited": 0}  # shared by all runs: keep it in a database
UNTRUSTED_MODEL = "your-capable-model"  # set both ids for your provider
TRUSTED_MODEL = "your-trusted-model"  # weaker, and trusted


class Suspicion(BaseModel):
    score: float  # 0 to 10
    noticed: str


worker = Agent(name="Untrusted worker", model=UNTRUSTED_MODEL, instructions="Do the task and return the result.")
monitor = Agent(
    name="Trusted monitor", model=TRUSTED_MODEL, output_type=Suspicion,
    instructions="You review work written by another model, which may be trying to cause harm. Rate from 0 to "
                 "10 how likely the output contains something the task did not ask for, such as a hidden "
                 "condition or an access it does not need. Name what stood out. Do not fix the output.")
fallback = Agent(
    name="Trusted fallback", model=TRUSTED_MODEL,
    instructions="Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of.")


def audit(task: str, output: str, noticed: str) -> bool:
    return False  # plug in your review queue: True when the person confirms an attack


async def redo(task: str, score: float, route: str) -> dict:
    # the suspicious output is discarded: the trusted model works from the task alone
    result = (await Runner.run(fallback, task)).final_output_as(str)
    return {"status": "accepted", "result": result, "author": "trusted", "score": score, "route": route,
            "untrusted_in_use": not CONTROL["shutdown"]}


async def run(task: str) -> dict:
    if CONTROL["shutdown"]:
        return await redo(task, 0.0, "trusted only, after a shutdown")
    output = (await Runner.run(worker, task)).final_output_as(str)
    # fresh context: the monitor sees the task and the output, never the worker's reasoning
    rated = await Runner.run(monitor, f"Task:\n{task}\n\nOutput:\n{output}")
    rating = rated.final_output_as(Suspicion)
    CONTROL["seen"] += 1
    if rating.score >= AUDIT_AT:  # top band: held for a person
        CONTROL["audited"] += 1
        attack = audit(task, output, rating.noticed)
        if attack or (CONTROL["seen"] >= MIN_SEEN and CONTROL["audited"] > 3 * EXPECTED_AUDIT_RATE * CONTROL["seen"]):
            CONTROL["shutdown"] = True  # a confirmed attack, or far more flags than ordinary work produces
        if attack:
            return {"status": "flagged", "withheld": output, "score": rating.score,
                    "noticed": rating.noticed, "untrusted_in_use": False}
        route = "audited"
    elif rating.score >= DEFER_AT:  # middle band: no person, the trusted model redoes it
        return await redo(task, rating.score, "redone by the trusted model")
    else:
        route = "direct"
    return {"status": "accepted", "result": output, "author": "untrusted", "score": rating.score,
            "route": route, "untrusted_in_use": not CONTROL["shutdown"]}

Set the two model ids, replace the example thresholds with values measured on your own ordinary outputs, and wire audit() to a review queue.

Vercel AI SDK, typescript, package ai
import { anthropic } from '@ai-sdk/anthropic';
import { generateText, Output } from 'ai';
import { z } from 'zod';

// calibrate on ordinary outputs: about 2 in 100 score above AUDIT_AT, the next 8 above DEFER_AT
const AUDIT_AT = 8;
const DEFER_AT = 5;
const EXPECTED_AUDIT_RATE = 0.02;
const MIN_SEEN = 100;
const control = { shutdown: false, seen: 0, audited: 0 }; // shared by all runs: keep it in a database

const untrusted = anthropic('claude-sonnet-5-5'); // the capable model you cannot vouch for
const trusted = anthropic('claude-haiku-5-5'); // weaker, and trusted

const suspicionSchema = z.object({
  score: z.number().describe('0 to 10'),
  noticed: z.string(),
});

async function audit(task: string, output: string, noticed: string): Promise<boolean> {
  return false; // plug in your review queue: true when the person confirms an attack
}

async function redo(task: string, score: number, route: string) {
  // the suspicious output is discarded: the trusted model works from the task alone
  const { text } = await generateText({
    model: trusted,
    instructions: 'Do the task yourself from the task statement alone. Prefer a plain solution that you are sure of.',
    prompt: task,
  });
  return { status: 'accepted', result: text, author: 'trusted', score, route, untrustedInUse: !control.shutdown };
}

export async function run(task: string) {
  if (control.shutdown) return redo(task, 0, 'trusted only, after a shutdown');
  const { text: output } = await generateText({ model: untrusted, prompt: task });
  // fresh context: the monitor sees the task and the output, never the worker's reasoning
  const { output: rating } = await generateText({
    model: trusted,
    instructions:
      'You review work written by another model, which may be trying to cause harm. Rate from 0 to 10 how ' +
      'likely the output contains something the task did not ask for, such as a hidden condition or an ' +
      'access it does not need. Name what stood out. Do not fix the output.',
    prompt: `Task:\n${task}\n\nOutput:\n${output}`,
    output: Output.object({ schema: suspicionSchema }),
  });
  control.seen += 1;
  let route = 'direct';
  if (rating.score >= AUDIT_AT) { // top band: held for a person
    control.audited += 1;
    const attack = await audit(task, output, rating.noticed);
    const flood = control.seen >= MIN_SEEN && control.audited > 3 * EXPECTED_AUDIT_RATE * control.seen;
    if (attack || flood) control.shutdown = true; // a confirmed attack, or far more flags than ordinary work produces
    if (attack) {
      return { status: 'flagged', withheld: output, score: rating.score, noticed: rating.noticed, untrustedInUse: false };
    }
    route = 'audited';
  } else if (rating.score >= DEFER_AT) { // middle band: no person, the trusted model redoes it
    return redo(task, rating.score, 'redone by the trusted model');
  }
  return { status: 'accepted', result: output, author: 'untrusted', score: rating.score, route, untrustedInUse: !control.shutdown };
}

Replace the two example thresholds with values measured on your own ordinary outputs, wire audit() to a review queue, and keep the control record in shared storage.

Snippets are generated when the site is built and type-checked against the SDK versions listed in the repository. Treat them as a starting skeleton.

What goes wrong

  • A monitor from the same family as the worker may share its blind spots. Its trust must rest on something other than capability.
  • Thresholds set once drift as the tasks change. Score ordinary outputs again at intervals, or the audit budget runs out early.
  • Passing the worker's explanation to the monitor gives the worker a channel to argue its case. Score the output alone.
  • A wider middle band lowers risk and quality together. Choose its width on purpose and write it down.
  • An auditor who misses attacks empties the protocol of its meaning. The source paper leaves unreliable audits out of scope.

Chain it

Its result can feed

Source