How Chao Uses Tools

Chao is not a static assistant that only knows what was loaded at session start. During a conversation, Chao can call tools to fetch live data from your account — bits, chainies, and calendar views — and use the results to give grounded, specific responses.

See also

• Explanation: Chao Context Model
• Reference: SSE Streaming
• How-to: Start a Chao Session

---

Why tool calling exists

Before tool calling, Chao received a static snapshot of your data at session start: a handful of recent bits, your active chainies, and memory entries. This worked for broad coaching ("how am I doing?") but fell short for specific questions ("show me everything overdue in my Morning Athlete goal" or "what do I have scheduled for next Tuesday?").

Tool calling lets Chao query your actual data in real time, on demand, as part of answering a question. The result is a Chao that can reason over your full dataset — not just a summary of it.

---

What happens when Chao uses a tool

From the client's perspective, a conversation turn with tool calls follows this sequence:

1. You send a message.
2. The platform may stream capability.resolved, plan.step_added, or tool.started events depending on the request.
3. Each tool resolves and the platform streams tool.progress, tool.completed, tool.failed, or tool.degraded.
4. Chao receives the results internally, may decide to call additional tools, and eventually writes its response.
5. The response streams token by token via message.delta, then closes with message.completed.

Tool events may include sanitized observation data for card rendering. Chao's response is still the user-facing synthesis.

Chao now uses an agent loop, so a single user message can produce multiple sequential tool.started / tool.completed cycles before message.completed arrives. If your client subscribes to the optional agent.phase.changed event, you can render the high-level phase the agent is in (planning, executing a tool, reading the result, verifying the answer). See AI Run Lifecycle → Agent-style Runs for the full event reference.

When Chao accidentally asks for the same read tool with the same input again in the same run, the platform reuses the earlier observation instead of rerunning the tool. That keeps replies faster and avoids duplicate internal tool charges.

Chao requires structured tool calls to read or modify your workspace data — narrating an action in prose (e.g. "I need to fetch your active bits…") is not enough. If the model describes an action without actually calling the tool, the platform retries once with a stronger instruction before responding, and may widen the available tool catalog to include the data Chao mentioned. You usually never see the discarded prose; only the final grounded answer arrives at your client.

SSE event sequence example

tool.started        { toolKey: "calendar.overdue", callId: "tc_1", activity: { label: "Checking overdue tasks" } }
tool.completed      { toolKey: "calendar.overdue", callId: "tc_1", executionMs: 52, success: true }
message.delta       { delta: "You have 3 overdue items..." }
message.delta       { delta: " The oldest is..." }
message.completed   { content: "..." }
run.settlement.completed

---

What tools Chao has

Chao has built-in tools across productivity, memory, planning, research, and media domains:

Calendar tools (read)

Tool	What it does
calendar.today	All bits scheduled for today
calendar.week	Bits in the current, next, or previous week
calendar.range	Bits in any specific date range
calendar.overdue	All overdue pending bits

Bits tools (read + write)

Tool	What it does
bits.query	Filtered list of bits (date, status, priority, chainy, text, overdue)
bits.get	A single bit by ID
bits.search	Bits matching a text search
bits.create	Create a new bit (task). Title required, defaults: today + medium priority
bits.update	Update an existing bit (title, priority, scheduled date, etc.)
bits.complete	Mark a bit as completed, with optional completion note
bits.delete	Soft-delete a bit (hidden from queries, not permanently removed)

Chainies tools (read + write)

Tool	What it does
chainies.list	Your goal systems, optionally filtered by status
chainies.get	A single chainy by ID
chainies.search	Chainies matching a text search
chainies.create	Create a new chainy (goal). Title required.

Chains tools (read + write)

Tool	What it does
chains.list	Your habits/routines, optionally filtered by chainy or status
chains.get	A single chain by ID with streak data
chains.create	Create a new chain linked to a chainy

Memory tools

Tool	What it does
memory.search	Search your stored memories by text, type, or chainy scope
memory.store	Store a new memory — facts, preferences, goals, or context

Models tool

Tool	What it does
models.search	List available AI models, filterable by provider or capability

Planning tools

Tool	What it does
planning.generate	Generate a structured multi-step plan for a goal (rate limited: 3/hour)
planning.list	List your AI-generated plans, filterable by status or linked goal
planning.get	Get a specific plan including all steps

Search & research tools

Tool	What it does
search.web	Search the web across up to 3 queries, deduplicate results, and return an optional synthesized answer
research.deep	Deep research on a topic — synthesizes multiple sources into a summary with citations

Media capability tools

Public capability	Tool Chao runs	What it does
ai.image.generate	media.image.generate	Generate, transform, recolor, restyle, or edit an image
ai.video.generate	media.video.generate	Generate a video (see Video parameters below)
ai.audio.generate	media.audio.generate	Generate a short music clip via Google Lyria

Clients can keep displaying the public capability key (ai.image.generate) for product and entitlement UI. Live tool cards should accept the execution tool key (media.image.generate).

When sending capability-specific parameters with a message, the request goes through a two-phase validator: the common envelope is validated first, then a capability-aware parameter spec checks the per-capability fields. Adding a future capability requires only a new *ParameterSpec class — the message DTO and controller stay untouched. Wrong-capability fields produce precise errors (bpm is only valid for ai.audio.generate) instead of generic whitelist errors. See Messages: Capability Parameters for the full per-capability field list and the nested-vs-top-level resolution rule.

Audio parameters

media.audio.generate accepts optional inputs alongside the required prompt:

Input	Type	Default	Notes
durationSeconds	number	20	Clamped to 30 s max; ignored if ≤ 0
bpm	number	—	Clamped to 40–220; omitted if missing or ≤ 0
styleTokens	string[]	—	Genre, instrument, or mood hints (e.g. ["jazz", "piano", "upbeat"])

These parameters are also accepted by the direct REST endpoint POST /ai/generative-media/context-runs when mediaType is "audio".

Video parameters

media.video.generate accepts { prompt } only — all other fields are optional and may be omitted to let the selected video model infer the best output. When provided, they override the model and are forwarded to the provider after normalization.

Publicly surfaced fields for UI pickers:

Field	Values	Notes
prompt	string	Required
provider	kling, veo, sora	Defaults to the configured provider
aspectRatio	16:9, 9:16, 1:1	Provider may reject ratios it doesn't support; 1:1 is only honored by Kling
duration	number (seconds)	Subject to provider limits (Sora ≤ 20, Veo 1080p = 8, Kling ∈ {5, 10})

Additional optional fields exist for power users — size, resolution, negativePrompt, seed, generateAudio, inputImageUrl, referenceImages, referenceVideos, styleTokens — and are forwarded to the provider when supported, ignored otherwise.

---

Intermediate progress events

For long-running tools, Chao publishes tool.progress events between tool.started and tool.completed. These let you show specific status messages in your UI instead of a generic spinner.

SSE event sequence with progress

tool.started    { toolKey: "bits.search", callId: "..." }
tool.progress   { toolKey: "bits.search", message: "Searching your tasks..." }
tool.progress   { toolKey: "bits.search", message: "Found 12 matching tasks" }
tool.completed  { toolKey: "bits.search", executionMs: 110, success: true }

tool.progress events are informational — they do not affect the final result. If a progress event is missed or delayed, the tool still completes normally.

Multi-step web search progress

When search.web runs with multiple queries it publishes one tool.progress per step, so your UI can show exactly what Chao is doing:

tool.started    { toolKey: "search.web", callId: "..." }
tool.progress   { toolKey: "search.web", message: "Understanding your search query..." }
tool.progress   { toolKey: "search.web", message: "Executing 3 searches for comprehensive coverage..." }
tool.progress   { toolKey: "search.web", message: "Searching the web... \"Claude AI capabilities\"" }
tool.progress   { toolKey: "search.web", message: "Searching the web... \"Anthropic safety research\"" }
tool.progress   { toolKey: "search.web", message: "Searching the web... \"LLM benchmarks 2025\"" }
tool.progress   { toolKey: "search.web", message: "Analyzing search results..." }
tool.progress   { toolKey: "search.web", message: "Found 18 results" }
tool.progress   { toolKey: "search.web", message: "Synthesizing findings..." }
tool.completed  { toolKey: "search.web", executionMs: 4210, success: true }

Chao selects whether to use a single query or multiple based on the complexity of your request. When multiple queries run, the results are automatically deduplicated by URL before being returned.

---

Write tools

Chao can now create, update, complete, and delete your bits, create chainies and chains, and store memories. All write operations:

• Verify ownership before mutating — Chao can only modify data that belongs to your account.
• Use soft deletes — deleted bits are hidden, not permanently removed.
• Report results naturally — Chao confirms what it did in its response.
• Are rate-limited — maximum 30 tool calls per minute per session to prevent runaway loops.

Example: "Create a task to review the Q2 report by Friday"

tool.started        { toolKey: "bits.create" }
tool.completed      { toolKey: "bits.create", executionMs: 35, success: true }
message.delta       { delta: "Done! I created a task..." }
message.completed   { content: "..." }

---

Session modes: controlling how Chao executes write tools

Every Chao session has a mode that controls whether write tools execute immediately or require your involvement. You can change the mode when creating a session (mode field on POST /ai/sessions) or at any time during a session (PATCH /ai/sessions/:id).

Mode	Behavior
approval (default)	Chao proposes each write action and waits for you to approve or reject before executing. Safe default — nothing changes without your explicit sign-off.
auto	Chao executes write tools immediately without asking. Use this when you want Chao to act without interruption.
plan	Chao collects write actions into a plan instead of executing them. When you're ready, review the plan and run it all at once with POST /ai/plans/:planId/execute.

Read-only tools (calendar, search, list, get) are never gated — they execute immediately in all modes.

---

Approval mode

When Chao wants to run a write tool in approval mode:

1. The platform streams tool.approval_required with the tool name and arguments.
2. Your client shows a confirmation prompt.
3. You call POST /ai/runs/:runId/tools/:toolCallId/approve or .../reject.
4. If approved, the tool runs and Chao confirms the result. If rejected, Chao acknowledges.

If no response arrives within 60 seconds, the call is automatically rejected and Chao explains the timeout.

SSE event sequence (approval mode):

tool.approval_required   { toolKey: "bits.create", toolCallId: "...", input: { title: "..." } }
tool.approval_response   { toolCallId: "...", approved: true }
tool.started             { toolKey: "bits.create" }
tool.completed           { toolKey: "bits.create", executionMs: 35 }
message.delta            { delta: "Done! I created..." }

Approve or reject with:

• POST /ai/runs/:runId/tools/:toolCallId/approve
• POST /ai/runs/:runId/tools/:toolCallId/reject

---

Plan mode

plan mode routes the message to a dedicated Planning Agent instead of the normal chat / tool loop. Use it when you want a structured implementation plan rather than the assistant doing the work itself.

The plan is streamed back as a markdown document on the standard message.delta / message.completed events — no special event types are needed. The output always contains these eight top-level sections, in order:

# Implementation Plan
## Overview
## Requirements
## Architecture Changes
## Implementation Steps
## Testing Strategy
## Risks & Mitigations
## Success Criteria

Tool policy in plan mode: write tools are unavailable — the dispatcher blocks them and returns { blocked: true, reason: "plan_mode_read_only" }. Read-only tools (search, list, get) still run so the planner can ground its plan in actual data. If the model tries to call a write tool anyway, the client sees a tool.failed event with that payload and the planner continues without executing the action.

SSE event sequence (plan mode):

run.started       { runId: "..." }
message.delta     { content: "# Implementation Plan\n\n## Overview\n..." }
message.delta     { content: "...\n\n## Implementation Steps\n..." }
message.completed { content: "<full plan markdown>" }
run.settlement.completed

To turn plan mode on or off for a session:

PATCH /ai/sessions/:sessionId
{ "mode": "plan" }   // or "approval" / "auto" to switch back

---

Plan-collect mode (legacy "deferred writes")

plan_collect is the older "build a plan of tool calls and execute them later" behaviour. The conversation runs normally, but write tools are captured into an ai_plans record instead of being executed in-band:

1. Each write tool call becomes a step in the session's plan.
2. The platform streams plan.step_added for each collected step.
3. When Chao has finished building the plan, you see an ordered list of actions to review.
4. Call POST /ai/plans/:planId/execute to run all steps in sequence.

SSE event sequence — conversational write tools (plan_collect mode):

plan.step_added   { planId: "...", step: { order: 1, toolKey: "bits.create", description: "Create bit: Morning run" } }
plan.step_added   { planId: "...", step: { order: 2, toolKey: "bits.create", description: "Create bit: Evening workout" } }
message.completed { content: "I've added 2 tasks to your plan. Review and run when ready." }

SSE event sequence — capability tools in plan_collect mode (image generation, video generation, web search, deep research):

When a capability tool is selected and the session is in plan_collect mode, the capability is queued as a plan step rather than executed immediately. The assistant message always contains a human-readable explanation:

tool.started      { toolKey: "media.image.generate", callId: "...", status: "executing" }
plan.step_added   { planId: "...", step: { order: 1, toolKey: "media.image.generate", description: "Generate image: ..." } }
tool.completed    { toolKey: "media.image.generate", callId: "...", data: { addedToPlan: true, planId: "..." } }
message.completed { content: "I've added image generation to your plan: \"...\". Review the plan and click Execute when you're ready." }
run.settlement.completed

The tool call on the assistant message carries output.addedToPlan: true. Render such tool calls with an "In plan" indicator rather than a success state — the tool has not executed yet.

Executing a plan:

• GET /ai/plans?sessionId=<id> — fetch the draft plan for the session
• POST /ai/plans/:planId/execute — run all pending steps in order
• DELETE /ai/plans/:planId — discard the plan without executing

Tools that support plan_collect mode: bits.create, bits.update, bits.delete, bits.complete, chainies.create, chains.create, memory.store, planning.generate, media.image.generate, media.video.generate, media.audio.generate, search.web, research.deep

---

---

Semantic memory search

When you store memories ("remember that I prefer bullet points"), Chao generates an embedding — a numerical representation of the meaning — and stores it alongside the text.

When you later ask something conceptually related ("what are my formatting preferences?"), Chao searches by meaning, not just keywords. This means:

• "I like concise responses" matches a search for "communication style"
• "My morning routine starts at 6am" matches "daily schedule"
• "I'm training for a half marathon" matches "fitness goals"

Chao combines semantic similarity (meaning-based) with keyword matching to find the most relevant memories. If no embeddings exist yet (e.g., you just started using memories), Chao falls back to keyword search with no disruption.

---

Data access and privacy

All tool calls are scoped to your account. Chao cannot read data from other accounts. Every query runs as your account, under the same authorization rules as if you had called the API directly.

Tool results are passed to Chao's response context and may be persisted on the assistant message so clients can reconstruct tool cards after reload. They are not stored as long-term memories automatically.

---

How Chao decides to use tools

Chao decides autonomously, based on the question, whether to call tools. It first checks whether the current conversation already has enough context, then selects the smallest useful tool set only when live account data, external information, or an explicit action is required.

If the information Chao needs is already in the session context (loaded at session start), Chao will use it without calling a tool. Tool calls only happen when the question requires data that isn't in the initial snapshot.

Chao may call multiple tools in a single turn when the request genuinely needs them, or chain tool calls across multiple iterations if the first round of results prompts a follow-up query. It should not call tools simply because they are available.

---

Pagination and large datasets

Chao uses keyset pagination when fetching large lists. If there are more results than a single tool call returns, Chao can fetch subsequent pages by passing the cursor from the previous response. This happens automatically within a single conversation turn.

Chao applies sensible defaults (20 items per page, max 50) to stay within token budget. For very large datasets, Chao may summarize rather than enumerate every item.

---

Frontend integration (ViteJS / TypeScript)

Use SSE tool events to show "Chao is working..." indicators. Here is a minimal example using the browser EventSource API:

const eventSource = new EventSource(
  `${BASE_URL}/ai/sessions/${sessionId}/messages/${messageId}/stream`,
  { headers: { Authorization: `Bearer ${token}` } }
);

const toolCards = new Map<string, {
  toolKey?: string;
  input?: unknown;
  output?: unknown;
  activity?: {
    activityType?: string;
    label?: string;
    detail?: string;
    renderHint?: string;
    status?: string;
    subjectType?: string;
    subjectId?: string;
  };
}>();

const normalizeToolPayload = (envelope: any) => {
  const payload = envelope.payload ?? {};
  return {
    toolKey: payload.toolKey ?? payload.key,
    callId: payload.callId ?? payload.toolCallId ?? envelope.stepId,
    input: payload.input ?? payload.args,
    output: payload.data ?? payload.output,
    activity: payload.activity,
  };
};

const upsertToolCard = (e: MessageEvent) => {
  const envelope = JSON.parse(e.data);
  const card = normalizeToolPayload(envelope);
  if (!card.callId) return;
  toolCards.set(card.callId, card);
  renderToolCard(card);
};

[
  'capability.resolved',
  'plan.step_added',
  'tool.started',
  'tool.progress',
  'tool.completed',
  'tool.failed',
  'tool.degraded',
  'tool.approval_required',
  'tool.approval_response',
  'tool.approval_timeout',
].forEach((type) => eventSource.addEventListener(type, upsertToolCard));

eventSource.addEventListener('message.delta', (e) => {
  const { payload } = JSON.parse(e.data);
  appendToMessage(payload.delta);
});

eventSource.addEventListener('run.settlement.completed', () => {
  eventSource.close();
});

Use persisted message.toolCalls to rebuild the same cards after a page reload. tool.failed is not terminal; keep listening until message.completed, message.partially_completed, run.failed, run.error, or run.settlement.completed.

Tool indicator display names

Map tool keys to user-friendly labels for your UI:

const TOOL_LABELS: Record<string, string> = {
  'calendar.today':    'Checking today\'s schedule',
  'calendar.week':     'Looking at this week',
  'calendar.overdue':  'Finding overdue tasks',
  'bits.query':        'Searching your tasks',
  'bits.create':       'Creating a task',
  'bits.update':       'Updating a task',
  'bits.complete':     'Completing a task',
  'bits.delete':       'Removing a task',
  'chainies.list':     'Loading your goals',
  'chainies.create':   'Creating a goal',
  'chains.list':       'Loading your habits',
  'chains.create':     'Creating a habit',
  'memory.search':     'Searching memories',
  'memory.store':      'Saving a memory',
  'models.search':     'Checking available models',
  'planning.generate': 'Generating your plan',
  'planning.list':     'Loading your plans',
  'planning.get':      'Loading plan details',
  'search.web':         'Searching the web',
  'research.deep':      'Researching deeply',
  'media.image.generate': 'Generating an image',
  'media.video.generate': 'Generating a video',
  'media.audio.generate': 'Generating audio',
};

---

Rate limiting

Chao is limited to 30 tool calls per minute per session. This prevents runaway loops where Chao might call tools excessively. If the limit is reached, Chao will inform you and continue the conversation without additional tool calls.

---

Model compatibility

Tool calling requires a compatible AI model. The following providers support tool calling:

Provider	Tool Calling
Anthropic (Claude)	Supported
OpenAI (GPT-4)	Supported
Google (Gemini)	Supported
Mistral	Supported
Ollama (local models)	Not supported

If your session uses a model that does not support tool calling, Chao automatically operates in conversational mode using only the data loaded at session start. No error is raised — tools are simply unavailable.