LLM Integration Guide

This guide covers integrating Large Language Models (LLMs) with the Firefly Catcher Framework.

Overview

FFC provides a flexible LLM integration system that supports: - Multiple LLM providers - Template-based prompt management - Context handling - Token management - Error handling and fallback strategies

Provider Configuration

Supported Providers

• OpenAI
• Anthropic
• Custom providers

Configuration Example

from ffc.llm import LLMConfig, Provider

config = LLMConfig(
    provider=Provider.OPENAI,
    model="gpt-4",
    api_key="your-api-key",
    max_tokens=2000
)

Prompt Management

Template System

from ffc.llm import PromptTemplate

template = PromptTemplate(
    name="task_decomposition",
    content="Break down the following task into steps: {task}",
    variables=["task"]
)

Context Handling

from ffc.llm import Context

context = Context()
context.add("system_prompt", "You are a helpful AI assistant")
context.add("user_context", user_data)

Error Handling

Retry Mechanism

from ffc.llm import RetryConfig

retry_config = RetryConfig(
    max_retries=3,
    backoff_factor=1.5,
    errors_to_retry=[RateLimitError, TemporaryError]
)

Fallback Strategy

from ffc.llm import FallbackConfig

fallback_config = FallbackConfig(
    providers=[Provider.OPENAI, Provider.ANTHROPIC],
    models=["gpt-4", "claude-2"]
)

Cost Management

Token Tracking

from ffc.llm import TokenTracker

tracker = TokenTracker()
tracker.start_session()
# ... LLM operations ...
usage = tracker.get_session_usage()

Budget Control

from ffc.llm import BudgetManager

budget = BudgetManager(
    max_daily_tokens=100000,
    max_cost_usd=10.0
)

Best Practices

1. Token Efficiency
2. Use efficient prompts
3. Implement context windowing

4. Clean and preprocess inputs

5. Error Handling

6. Always implement retry logic
7. Use fallback providers

8. Log and monitor errors

9. Cost Control

10. Set budget limits
11. Monitor token usage

12. Implement cost optimization strategies

13. Security

14. Secure API key storage
15. Sanitize inputs
16. Implement rate limiting

Example Implementation

from ffc.llm import LLMManager, PromptTemplate, Context

# Initialize LLM manager
llm = LLMManager(config)

# Create prompt template
template = PromptTemplate(
    name="analysis",
    content="Analyze the following code: {code}"
)

# Prepare context
context = Context()
context.add("code", source_code)

# Execute LLM call with retry and fallback
result = await llm.generate(
    template=template,
    context=context,
    retry_config=retry_config,
    fallback_config=fallback_config
)