Best Practices for Production Deployments

[saschabuehrle]

Planning to deploy cascadeflow in production? Here are some best practices to ensure success!

Model Selection

Recommended: Claude-3.5-Haiku + GPT-5

• Haiku: Fast, high-quality drafts
• GPT-5: Superior reasoning for complex queries
• 50-65% cost savings on average
• 75% draft acceptance rate

Alternative: GPT-4o-mini + GPT-5

• Good balance of cost and quality
• 50-60% cost savings
• Best for OpenAI-only shops

Quality Threshold Tuning

Start with quality_threshold=0.7 and adjust based on your use case:

• 0.6-0.7: Higher draft acceptance, slightly lower quality
• 0.7-0.8: Balanced (recommended for most use cases)
• 0.8-0.9: Lower draft acceptance, higher quality

Monitoring

Track these key metrics:

result = agent.run(query)
print(f"Model used: {result.metadata.model_used}")
print(f"Cost: ${result.cost:.6f}")
print(f"Draft accepted: {result.metadata.draft_accepted}")
print(f"Quality score: {result.metadata.quality_score}")

Error Handling

try:
    result = agent.run(query)
except Exception as e:
    # Fallback to single model
    result = openai_client.chat.completions.create(...)

Rate Limiting

Remember that cascadeflow makes 1-2 API calls per query:

• Draft model call (always)
• Verifier model call (only if draft fails)

Factor this into your rate limit calculations.

Cost Tracking

Use the built-in cost tracker:

from cascadeflow.telemetry import CostTracker

tracker = CostTracker()
agent = CascadeAgent(models=[...], cost_tracker=tracker)

# After running queries
summary = tracker.get_summary()
print(f"Total cost: ${summary['total_cost']:.2f}")
print(f"Savings: {summary['savings_percentage']:.1f}%")

Deployment Checklist

• Set API keys in environment variables
• Configure quality threshold for your use case
• Set up monitoring and alerting
• Implement error handling and fallbacks
• Test with production-like traffic
• Monitor draft acceptance rate
• Track cost savings over time

Questions?

Drop a comment below if you have questions about production deployments!