Tool Interface Analysis

Analyzes how agent frameworks model, register, and execute tools. This skill examines the tool abstraction layer, schema generation, built-in inventory, and error feedback mechanisms.

Distinction from harness-model-protocol

tool-interface-analysis	harness-model-protocol
How a "tool" is represented (types, base classes)	How tool calls are encoded on the wire
Schema generation (Pydantic -> JSON Schema)	Schema transmission to LLM API
Built-in tool inventory	Provider-specific tool formats
Registration and discovery patterns	Message format translation
Error feedback to LLM for retry	Response parsing and streaming
Tool execution orchestration	Partial tool call handling

Process

1. Map tool modeling - Identify how tools are represented (types, protocols, base classes)
2. Analyze schema generation - How tool definitions become JSON Schema
3. Catalog built-in inventory - What tools ship with the framework
4. Trace registration flow - How tools are discovered and made available
5. Document execution patterns - Invocation, validation, error handling
6. Evaluate retry mechanisms - Self-correction and feedback loops

Tool Modeling Patterns

Abstract Base Class Pattern

from abc import ABC, abstractmethod
from typing import Any

class BaseTool(ABC):
    """Framework's tool abstraction."""
    name: str
    description: str

    @abstractmethod
    def execute(self, **kwargs) -> Any:
        """Execute the tool with validated arguments."""
        ...

    @property
    @abstractmethod
    def parameters_schema(self) -> dict:
        """Return JSON Schema for parameters."""
        ...

Characteristics: Explicit contract, inheritance-based, type-safe Used by: LangChain, CrewAI, AutoGen

Protocol/Interface Pattern

from typing import Protocol, runtime_checkable

@runtime_checkable
class Tool(Protocol):
    """Structural typing for tools."""
    name: str
    description: str

    def __call__(self, **kwargs) -> Any: ...
    def get_schema(self) -> dict: ...

Characteristics: Duck typing, flexible, composition-friendly Used by: Pydantic-AI, OpenAI Agents SDK

Decorated Function Pattern

from functools import wraps

def tool(name: str = None, description: str = None):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            return func(*args, **kwargs)
        wrapper._tool_name = name or func.__name__
        wrapper._tool_description = description or func.__doc__
        wrapper._is_tool = True
        return wrapper
    return decorator

@tool(description="Search the web for information")
def search(query: str, max_results: int = 10) -> list[str]:
    ...

Characteristics: Lightweight, DRY, preserves function identity Used by: Google ADK, Swarm, Function calling patterns

Pydantic Model Pattern

from pydantic import BaseModel, Field

class SearchInput(BaseModel):
    """Input schema for search tool."""
    query: str = Field(description="The search query")
    max_results: int = Field(default=10, ge=1, le=100)

class SearchTool(BaseTool):
    name = "search"
    description = "Search the web"
    args_schema = SearchInput

    def execute(self, **kwargs) -> list[str]:
        validated = SearchInput(**kwargs)
        return perform_search(validated.query, validated.max_results)

Characteristics: Rich validation, auto-schema, clear separation Used by: LangChain, CrewAI

Schema Generation Methods

Introspection-Based (Automatic)

import inspect
from typing import get_type_hints

def generate_schema_from_function(func) -> dict:
    hints = get_type_hints(func)
    sig = inspect.signature(func)
    doc = inspect.getdoc(func) or ""

    schema = {
        "type": "function",
        "function": {
            "name": func.__name__,
            "description": doc.split("\n")[0],
            "parameters": {
                "type": "object",
                "properties": {},
                "required": []
            }
        }
    }

    for name, param in sig.parameters.items():
        if name in ("self", "cls"):
            continue

        prop = {"type": python_type_to_json(hints.get(name, str))}

        # Extract description from docstring if available
        if f":param {name}:" in doc:
            prop["description"] = extract_param_doc(doc, name)

        if param.default is inspect.Parameter.empty:
            schema["function"]["parameters"]["required"].append(name)
        else:
            prop["default"] = param.default

        schema["function"]["parameters"]["properties"][name] = prop

    return schema

Pros: DRY, always in sync with code, minimal boilerplate Cons: Limited expressiveness, relies on annotations, docstring parsing fragile

Pydantic-Based (Semi-Automatic)

from pydantic import BaseModel, Field
from pydantic.json_schema import GenerateJsonSchema

class SearchInput(BaseModel):
    """Search the web for information."""
    query: str = Field(description="The search query")
    max_results: int = Field(default=10, ge=1, le=100, description="Max results to return")

def generate_schema_from_pydantic(model: type[BaseModel]) -> dict:
    return {
        "type": "function",
        "function": {
            "name": model.__name__.replace("Input", "").lower(),
            "description": model.__doc__ or "",
            "parameters": model.model_json_schema()
        }
    }

Pros: Rich validation, excellent descriptions, composable Cons: Class per tool, more boilerplate, Pydantic dependency

Decorator-Based (Explicit)

@tool(
    name="search",
    description="Search the web for information",
    parameters={
        "query": {"type": "string", "description": "Search query"},
        "max_results": {"type": "integer", "default": 10}
    }
)
def search(query: str, max_results: int = 10) -> list[str]:
    ...

Pros: Explicit, flexible, no dependencies Cons: Can drift from implementation, manual maintenance

Manual Definition

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search",
            "description": "Search the web for information",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The search query"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

Pros: Full control, no magic, portable Cons: Maintenance burden, drift risk, duplication

Schema Generation Comparison

Method	Sync with Code	Expressiveness	Boilerplate	Validation
Introspection	Automatic	Low	None	None
Pydantic	Automatic	High	Medium	Built-in
Decorator	Manual	Medium	Low	Optional
Manual	Manual	Full	High	None

Registration Patterns

Declarative List

agent = Agent(
    tools=[search_tool, calculator_tool, weather_tool]
)

Characteristics: Explicit, static, easy to understand, testable

Registry Pattern

TOOL_REGISTRY = {}

def register_tool(name: str = None):
    def decorator(func):
        tool_name = name or func.__name__
        TOOL_REGISTRY[tool_name] = func
        return func
    return decorator

@register_tool("search")
def search(query: str) -> list[str]: ...

# Agent uses registry
agent = Agent(tools=list(TOOL_REGISTRY.values()))

Characteristics: Dynamic, plugin-friendly, implicit coupling

Discovery-Based (Auto-Import)

import importlib
import pkgutil

def discover_tools(package):
    tools = []
    for module_info in pkgutil.iter_modules(package.__path__):
        module = importlib.import_module(f"{package.__name__}.{module_info.name}")
        for name, obj in inspect.getmembers(module):
            if hasattr(obj, '__tool__') or isinstance(obj, BaseTool):
                tools.append(obj)
    return tools

# Usage
from myapp import tools as tools_package
agent = Agent(tools=discover_tools(tools_package))

Characteristics: Automatic, magic, harder to trace, good for plugins

Factory Pattern

class ToolFactory:
    _registry: dict[str, type[BaseTool]] = {}

    @classmethod
    def register(cls, name: str):
        def decorator(tool_class):
            cls._registry[name] = tool_class
            return tool_class
        return decorator

    @classmethod
    def create(cls, config: ToolConfig) -> BaseTool:
        tool_class = cls._registry.get(config.type)
        if not tool_class:
            raise ValueError(f"Unknown tool type: {config.type}")
        return tool_class(**config.params)

# Registration
@ToolFactory.register("search")
class SearchTool(BaseTool): ...

# Creation
tool = ToolFactory.create(ToolConfig(type="search", params={"api_key": "..."}))

Characteristics: Configurable, testable, DI-friendly, more complex

Toolset/Toolkit Pattern

class WebToolkit:
    """Collection of related tools."""

    def __init__(self, api_key: str):
        self.api_key = api_key

    def get_tools(self) -> list[BaseTool]:
        return [
            SearchTool(api_key=self.api_key),
            BrowseTool(api_key=self.api_key),
            ExtractTool(api_key=self.api_key)
        ]

# Usage
agent = Agent(tools=WebToolkit(api_key="...").get_tools())

Characteristics: Cohesive grouping, shared configuration, composable

Error Feedback Analysis

Feedback Quality Levels

Level	What LLM Sees	Self-Correction Possible
Silent	Nothing	No
Basic	Exception type	Unlikely
Message	Exception message	Sometimes
Detailed	Type + message + context	Often
Structured	Error object with hints	Yes
Actionable	Suggestion + example	Very likely

Implementation Patterns

Silent (Anti-Pattern)

def run_tool(self, tool, args):
    try:
        return tool.execute(**args)
    except Exception:
        return None  # Error lost - LLM has no feedback

Basic

def run_tool(self, tool, args):
    try:
        return tool.execute(**args)
    except Exception as e:
        return f"Error: {type(e).__name__}"

Detailed with Context

@dataclass
class ToolResult:
    success: bool
    output: Any = None
    error: str | None = None
    error_type: str | None = None
    suggestion: str | None = None

def run_tool(self, tool, args) -> ToolResult:
    try:
        # Validate first
        validated = tool.validate_args(args)
        result = tool.execute(**validated)
        return ToolResult(success=True, output=result)
    except ValidationError as e:
        return ToolResult(
            success=False,
            error=str(e),
            error_type="validation_error",
            suggestion=f"Check parameter types: {e.errors()}"
        )
    except ToolExecutionError as e:
        return ToolResult(
            success=False,
            error=str(e),
            error_type="execution_error",
            suggestion=e.suggestion if hasattr(e, 'suggestion') else None
        )

Structured for LLM Consumption

def format_error_for_llm(self, result: ToolResult) -> str:
    if result.success:
        return str(result.output)

    parts = [f"Tool execution failed: {result.error}"]

    if result.error_type == "validation_error":
        parts.append("The provided arguments did not match the expected schema.")

    if result.suggestion:
        parts.append(f"Suggestion: {result.suggestion}")

    return "\n".join(parts)

Retry Mechanisms

Simple Retry with Backoff

async def run_with_retry(self, tool, args, max_retries=3):
    for attempt in range(max_retries):
        result = await self.run_tool(tool, args)
        if result.success:
            return result
        if not self._is_retryable(result.error_type):
            return result
        await asyncio.sleep(2 ** attempt)  # Exponential backoff
    return result

LLM-Guided Self-Correction

async def run_with_self_correction(self, tool, args, max_retries=3):
    for attempt in range(max_retries):
        result = await self.run_tool(tool, args)
        if result.success:
            return result

        # Ask LLM to fix the error
        correction_prompt = f"""
Tool `{tool.name}` failed with error: {result.error}
Original arguments: {json.dumps(args)}
Tool schema: {json.dumps(tool.parameters_schema)}

Provide corrected arguments as JSON.
"""
        corrected = await self.llm.generate(correction_prompt)
        args = json.loads(corrected)

    return result

Fallback Chain

async def run_with_fallback(self, tool_chain: list[BaseTool], args):
    for tool in tool_chain:
        result = await self.run_tool(tool, args)
        if result.success:
            return result
    return result  # Return last failure

Built-in Tool Categories

Common Categories

Category	Examples	Typical Implementation
Search	Web search, semantic search	API wrapper
Code	Execute code, REPL	Sandbox + subprocess
File	Read, write, list files	Filesystem API
Web	HTTP requests, scraping	HTTP client
Database	SQL query, vector search	Client + sanitization
Calculation	Math, unit conversion	Python eval or library
Memory	Store, retrieve facts	Vector store or KV
Communication	Email, Slack, API calls	API wrappers

Tool Inventory Template

Tool Name	Category	Input Schema	Output Type	Sandbox	Notes
search_web	Search	query: str	list[Result]	No	API key required
execute_python	Code	code: str	stdout: str	Yes	Isolated container
read_file	File	path: str	content: str	Partial	Path validation
http_request	Web	url, method, body	response	No	Rate limited

---

Output Document

When invoking this skill, produce a markdown document saved to:

forensics-output/frameworks/{framework}/phase2/tool-interface-analysis.md

Document Structure

The analysis document MUST follow this structure:

# Tool Interface Analysis: {Framework Name}

## Summary
- **Tool Modeling**: [Base class / Protocol / Decorated functions / Pydantic models]
- **Schema Generation**: [Introspection / Pydantic / Decorator / Manual]
- **Registration Pattern**: [Declarative / Registry / Discovery / Factory]
- **Error Handling**: [Silent / Basic / Detailed / Structured]
- **Built-in Tools**: [Count] tools in [N] categories

## Tool Modeling

### Core Abstraction

**Type**: [Abstract Base Class / Protocol / Decorated Function / Pydantic Model / Hybrid]

**Location**: `path/to/tool.py:L##`

```python
# Show the core tool type definition

Key Attributes: | Attribute | Type | Purpose | |-----------|------|---------| | name | str | Tool identifier for LLM | | description | str | Tool purpose for LLM selection | | parameters | ... | Input schema | | ... | ... | ... |

Inheritance/Composition:

BaseTool
├── BuiltinTool
├── APITool
└── CustomTool

Tool Creation Patterns

Pattern 1: [Name]

# Example code

Pattern 2: [Name] (if applicable)

# Example code

Schema Generation

Method Used

Primary Method: [Introspection / Pydantic / Decorator / Manual / Hybrid]

Location: path/to/schema.py:L##

Schema Generation Code

# Show how schemas are generated

Generated Schema Example

{
  "type": "function",
  "function": {
    "name": "example_tool",
    "description": "...",
    "parameters": {...}
  }
}

Type Mapping

Python Type	JSON Schema Type	Notes
str	string
int	integer
float	number
bool	boolean
list[T]	array	items type derived
dict	object
Optional[T]	T	Not in required
Union[A, B]	anyOf/oneOf

Built-in Tool Inventory

Tool Categories

Category	Tools	Purpose
Search	[list]	Information retrieval
Code	[list]	Code execution
File	[list]	File operations
...	...	...

Complete Tool List

Tool Name	Location	Schema Method	Category	Notes
tool_name	path:L##	Pydantic	Search	...
...	...	...	...	...

Tool Detail: [Example Tool]

Purpose: [What the tool does]

Input Schema:

# Show input type/schema

Output Type: [Return type]

Error Handling: [How errors are reported]

Registration & Discovery

Registration Pattern

Type: [Declarative List / Registry / Discovery / Factory / Toolkit]

Location: path/to/registration.py:L##

Registration Flow

1. Tool defined →
2. [Registration step] →
3. [Discovery step] →
4. Available to agent

Code Example

# Show registration code

Dynamic vs Static

• Static tools: [List or describe]
• Dynamic tools: [How tools are added at runtime, if supported]

Execution Flow

Invocation Pattern

Location: path/to/executor.py:L##

# Show tool execution code

Validation

Pre-execution validation: [Yes/No, method] Schema validation: [Pydantic / JSON Schema / Custom / None]

Error Handling

Error Type	Handling	Feedback to LLM
Validation error	...	...
Execution error	...	...
Timeout	...	...
Permission denied	...	...

Error Feedback Pattern

# Show how errors are formatted for LLM

Retry Mechanisms

• Automatic retry: [Yes/No, attempts, backoff]
• Self-correction: [Yes/No, LLM-guided]
• Fallback: [Yes/No, chain description]

Parallel Execution

Supported: [Yes/No]

Location: path/to/parallel.py:L##

Pattern: [Concurrent futures / asyncio.gather / Task groups]

# Show parallel execution code if present

Code References

• path/to/base_tool.py:L## - Core tool abstraction
• path/to/schema.py:L## - Schema generation
• path/to/registry.py:L## - Tool registration
• path/to/executor.py:L## - Tool execution
• path/to/builtin/*.py - Built-in tools
• ... (include all key file:line references)

Implications for New Framework

Positive Patterns

• Pattern 1: [Description and why to adopt]
• Pattern 2: [Description and why to adopt]
• ...

Considerations

• Trade-off 1: [Description]
• Trade-off 2: [Description]
• ...

Anti-Patterns Observed

• Anti-pattern 1: [Description and location]
• Anti-pattern 2: [Description and location]
• ...

---

Integration Points

• Prerequisite: codebase-mapping to identify tool-related files
• Related: harness-model-protocol for wire encoding of tool calls
• Related: resilience-analysis for error handling patterns
• Feeds into: comparative-matrix for interface decisions
• Feeds into: architecture-synthesis for tool layer design

Key Questions to Answer

1. How is a "tool" represented in this framework? (type, class, protocol)
2. How are tool schemas generated from definitions?
3. What built-in tools ship with the framework?
4. How are tools registered and discovered?
5. How is tool execution orchestrated?
6. How are errors fed back to the LLM for retry?
7. Does the framework support parallel tool execution?
8. How does validation work (pre-execution, schema-based)?
9. What retry/self-correction mechanisms exist?
10. Can tools be dynamically added/removed at runtime?

Files to Examine

When analyzing a framework, prioritize these file patterns:

Pattern	Purpose
**/tool*.py, **/tools/**	Tool definitions and base classes
**/schema*.py	Schema generation
**/registry*.py, **/register*.py	Tool registration
**/executor*.py, **/runner*.py	Tool execution
**/builtin*.py, **/default*.py	Built-in tool inventory
**/error*.py, **/exception*.py	Error types and handling
**/validation*.py	Argument validation
**/function*.py, **/callable*.py	Function-based tools