Engineering gaps: no circuit breaker, usage normalization accuracy, cache invalidation, rate limit strategy

[huiguyy]

Issue: Engineering Observations — Four Areas Needing Clarification or Enhancement

Hi, thanks for building Free-Way — it's a very useful tool for the free LLM ecosystem. After reviewing the codebase, I have four engineering observations that may benefit from documentation or implementation work:

1. No circuit breaker documented (高并发场景缺乏熔断机制)

The current automatic fallback strategy handles provider failures, but there is no documented circuit breaker or rate limiting behavior. In high-concurrency scenarios, if multiple providers are degraded simultaneously, the gateway may still fan out requests to unhealthy backends without throttling.

Suggested: Document the retry/fallback limits and consider adding a circuit breaker pattern (e.g., per-provider failure threshold → temporary open-circuit → health-check probe before reopening).

2. Usage normalization accuracy needs validation (Usage 归一化准确性需验证)

Different providers expose different field semantics in the usage object. The gateway normalizes usage at the gateway layer, but the mapping logic and accuracy of cross-provider normalization are not explicitly documented or tested.

Suggested: Add a USAGE_NORMALIZATION.md documenting:

• Which fields are mapped (input tokens, output tokens, total tokens)
• How each provider's native format is translated
• Known edge cases or unsupported fields

3. Model catalog cache invalidation not specified (模型目录缓存失效策略不明确)

The catalog uses a cache fallback mechanism, but the invalidation strategy (TTL, event-based refresh, manual-only) is not documented. This may lead to stale model lists being served after a provider removes or adds models.

Suggested: Document:

• Cache TTL (if any)
• How POST /api/models/refresh behaves
• Whether provider-level model availability changes trigger automatic re-fetch

4. No rate limit strategy documented (缺少 Provider 超限时的限流策略)

When a free-tier provider rate-limits or quota-exhausts, the current routing strategy is not specified. It's unclear whether the gateway:

• Retries with exponential backoff
• Immediately fails over to the next provider
• Returns the 429 directly to the client

Suggested: Document the rate-limit handling behavior, ideally with configurable strategies (e.g., retry vs. fail-fast per provider).

---

Overall Free-Way is excellent — these are not blockers but quality-of-documentation and production-hardening items. Happy to contribute a PR if the maintainers are open to it.

[GoDiao]

Hi @huiguyy,

Thank you for the detailed review — this is exactly the kind of feedback that helps a young project grow. All four points are valid and reflect real gaps between "works for personal use" and "production-grade gateway." Let me address each one with current status and intent:

1. Circuit breaker
You're right — right now we have naive retry (up to 3 attempts across providers) but no per-provider circuit breaker. If provider A starts timing out, we'll keep hitting it on every request until it happens to succeed. A simple failure-threshold → open-circuit → periodic probe pattern would go a long way.

2. Usage normalization accuracy
The normalization layer (src/usage.ts) currently does:

• Direct pass-through when the upstream returns standard OpenAI usage fields
• Conservative character-length estimation as fallback for providers that don't return usage
• Anthropic bridge converts input_tokens/output_tokens ↔ prompt_tokens/completion_tokens

It's intentionally conservative (never undercounts) but you're right that the mapping matrix isn't documented. A USAGE_NORMALIZATION.md would help users understand when numbers are exact vs. estimated.

3. Cache invalidation
Current behavior:

• Cache is loaded at startup from .freeway/*.json
• POST /api/models/refresh triggers a full re-sync across all providers
• No automatic TTL or event-based invalidation — stale lists persist until manual refresh or restart

This is definitely a "works for a dev laptop" approach, not a long-running server approach.

4. Rate limit handling
Today: 429s from upstream are treated as a failed attempt and trigger fallback to the next provider. No exponential backoff, no per-provider rate-limit tracking, no return of 429 to the client. It works for free-tier exploration but would be noisy under load.

---

Priority for me right now:

• (1) and (4) feel like the highest-impact production hardening items
• (2) is quick documentation win
• (3) needs a design decision (TTL vs. event-driven vs. lazy refresh)

If you're open to contributing, I'd love to see PRs for any of these — especially (2) since it's mostly documentation and would be a fast merge. For (1) and (4), happy to discuss design approach before you start coding so we align on scope.

Thanks again for taking the time to review the codebase so thoroughly. 🙏