fix(ai): add session affinity and compat fixes for Fireworks provider caching
Fireworks prompt caching is enabled by default (automatic prefix matching), but on serverless infrastructure, requests hit random replicas. Without session affinity, the per-replica cache misses, negating cache hit rates and the discounted cacheRead pricing. Changes: - Add sendSessionAffinityHeaders and supportsCacheControlOnTools to AnthropicMessagesCompat interface - Send x-session-affinity header for Fireworks (and Cloudflare AI Gateway Anthropic) when sessionId is available and caching is enabled - Omit cache_control on tool definitions for Fireworks (unsupported per https://docs.fireworks.ai/tools-sdks/anthropic-compatibility) - Default supportsEagerToolInputStreaming to false for Fireworks (unsupported field) - Default supportsLongCacheRetention to false for Fireworks (cache_control.ttl not supported) - Add compat settings to Fireworks models in generate-models.ts - Update generated models with Fireworks compat settings - Add integration tests for session affinity and tool compat Refs: https://docs.fireworks.ai/guides/prompt-caching Refs: https://docs.fireworks.ai/tools-sdks/anthropic-compatibility
This commit is contained in:
@@ -764,6 +764,16 @@ async function loadModelsDevData(): Promise<Model<any>[]> {
|
||||
},
|
||||
contextWindow: m.limit?.context || 4096,
|
||||
maxTokens: m.limit?.output || 4096,
|
||||
// Fireworks prompt caching uses automatic prefix matching + session affinity.
|
||||
// x-session-affinity routes requests to the same replica for cache hits.
|
||||
// cache_control on tools and eager_input_streaming are not supported.
|
||||
// See: https://docs.fireworks.ai/tools-sdks/anthropic-compatibility
|
||||
compat: {
|
||||
sendSessionAffinityHeaders: true,
|
||||
supportsEagerToolInputStreaming: false,
|
||||
supportsCacheControlOnTools: false,
|
||||
supportsLongCacheRetention: false,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user