Callback

Callback#

Understand what’s happening inside your agent with Synthora’s callback system. Callbacks provide a powerful way to log and track events throughout your agent’s workflow.

What Are Callbacks?#

Callbacks act as event listeners, providing updates on various stages of your agent’s operations. They notify you when:

  • The agent starts processing.

  • A tool is utilized.

  • Decisions are made.

  • An error occurs.

Adding Callbacks to Your Agent#

Implementing callbacks in your agent is simple. Below is an example of how to add logging with console output:

from synthora.agents import ReactAgent
from synthora.configs import AgentConfig
from synthora.callbacks import RichOutputHandler

# Create your agent
config = AgentConfig.from_file("examples/agents/configs/react_agent.yaml")
agent = ReactAgent.from_config(config)

# Add console output in two lines
handler = RichOutputHandler()
agent.add_handler(handler)

# Run the agent and observe the output
agent.run("Search OpenAI on Wikipedia. Output your thoughts first!")

Events You Can Track#

Callback handlers can monitor a wide range of events:

LLM Events#

  • on_llm_start: Triggered when the LLM begins processing input.

  • on_llm_end: Triggered when the LLM finishes generating a response.

  • on_llm_chunk: Provides real-time output during LLM streaming.

Tool Events#

  • on_tool_start: Triggered when a tool (e.g., search) starts executing.

  • on_tool_end: Triggered when a tool completes its task.

Agent Events#

  • on_agent_start: Triggered when the agent starts processing a task.

  • on_agent_end: Triggered when the agent concludes the task.

Asynchronous Callbacks#

To handle events asynchronously, use AsyncCallBackHandler. Here’s an example:

from synthora.callbacks import AsyncCallBackHandler

class MyAsyncHandler(AsyncCallBackHandler):
    async def on_llm_start(self, source, messages, **kwargs):
        print("Starting to process...")

    async def on_llm_end(self, source, message, **kwargs):
        print("Processing complete!")

Additional Features#

Add Multiple Handlers#

You can attach multiple handlers to your agent for different purposes:

agent.add_handler(RichOutputHandler())  # For console output
agent.add_handler(MyCustomHandler())    # For custom logging

Remove Handlers#

Handlers can be removed with ease:

agent.remove_handler(handler)
# Or simply use:
agent -= handler

Target Specific Components#

Handlers can also be added to specific tools within the agent:

# Add handler to the first tool in the agent's tool list
agent.tools[0].callback_manager.add(handler)

Quick Usage Example#

For a streamlined setup, you can initialize a default agent with handlers:

agent = VanillaAgent.default(ZeroShotCoTPrompt, handlers=[RichOutputHandler()])
agent.model.set_stream(True)

agent.run("How many letters 'r' are in the word 'strawberry'?")

Callback Event Reference Table#

Event Name

Trigger Timing

Parameters Description

on_llm_start

Triggered when the LLM starts processing input.

source: Node initiating the operation
messages: List of input messages
stream: Whether streaming is enabled

on_llm_end

Triggered when the LLM completes processing.

source: Node initiating the operation
message: Final output message
stream: Whether streaming was enabled

on_llm_error

Triggered when the LLM encounters an error.

source: Node where the error occurred
e: Exception raised
stream: Whether streaming was enabled

on_llm_chunk

Triggered when the LLM produces a streaming chunk.

source: Node producing the chunk
message: The chunked output message

on_tool_start

Triggered when a tool starts execution.

source: Node initiating the tool

on_tool_end

Triggered when a tool completes execution.

source: Node executing the tool
result: Result of the tool execution

on_tool_error

Triggered when a tool encounters an error.

source: Node where the error occurred
result: Error result from the tool

on_agent_start

Triggered when an agent starts processing.

source: Node initiating the agent
message: Initial message for the agent

on_agent_end

Triggered when an agent completes processing.

source: Node initiating the agent
message: Final message from the agent

on_agent_error

Triggered when an agent encounters an error.

source: Node where the error occurred
result: Error result from the agent

This table includes all the callback events and their associated parameters for effective tracking and debugging.

Callback Management with Managers#

Callbacks are managed using AsyncCallBackManager and BaseCallBackManager. These managers allow you to register and handle callback functions for various events, ensuring flexibility and control over how events are processed.

Key Points:#

  1. AsyncCallBackManager:

    • Supports both asynchronous and synchronous callback functions.

    • Provides the ability to handle events in asynchronous contexts.

  2. BaseCallBackManager:

    • Only supports synchronous callback functions.

    • Best suited for simple or non-asynchronous operations.

Summary of Differences:#

Manager

Supports Sync Callbacks

Supports Async Callbacks

AsyncCallBackManager

âś…

âś…

BaseCallBackManager

âś…

❌

By choosing the appropriate callback manager for your use case, you can ensure smooth event handling and proper integration with your agent’s workflow.

Summary#

Synthora’s callback system provides a powerful mechanism to track and manage events in your agent’s lifecycle. By leveraging callbacks, you gain deep insights into the agent’s operations, including LLM processing, tool usage, and overall task execution. Here’s a recap of the key takeaways:

  1. Comprehensive Event Tracking:

    • Monitor events such as LLM start, tool execution, agent tasks, and errors with detailed callbacks.

  2. Flexible Integration:

    • Use BaseCallBackHandler for synchronous callbacks or AsyncCallBackHandler for asynchronous workflows.

    • Easily customize behavior by implementing your own callback functions.

  3. Powerful Managers:

    • AsyncCallBackManager supports both synchronous and asynchronous callbacks.

    • BaseCallBackManager is restricted to synchronous callbacks, ensuring compatibility in simpler scenarios.

  4. Ease of Use:

    • Add or remove handlers effortlessly.

    • Target specific components like tools or agents for precise tracking.

By integrating Synthora’s callback system into your agents, you can enhance debugging, optimize performance, and create a more transparent and robust AI-driven system. Choose the right callback tools and managers to suit your project’s needs, and unlock the full potential of your agent workflows!

Contents