feat(amazon-bedrock): conditionally omit maxTokens from inferenceConfig (#3400)

When model.maxTokens is 0 (unset/unknown), omit maxTokens from the
Bedrock ConverseStream inferenceConfig instead of sending 0.

Bedrock's inferenceConfig.maxTokens is optional — when omitted, the
model uses its own internal default. This is optimal for Bedrock's
token quota management:

- At request start, Bedrock reserves input_tokens + max_tokens from
  your TPM quota
- For Claude 3.7+ models, output tokens have a 5x burndown rate
- Sending an unnecessarily high maxTokens wastes TPM capacity during
  the reservation window, reducing concurrent request throughput
- Omitting it lets Bedrock use the model default (~4096 for Claude),
  matching expected output sizes and maximizing quota utilization

Also applies the same conditional pattern to temperature — only
include it in inferenceConfig when explicitly set.

Changes:
- simple-options.ts: Use ?? instead of || so explicit 0 is preserved;
  return undefined when model.maxTokens is 0 (unset)
- amazon-bedrock.ts: Spread maxTokens and temperature into
  inferenceConfig only when defined

Fixes #3399
This commit is contained in:
wirjo
2026-04-19 16:48:43 +10:00
committed by GitHub
parent 5a889ef58b
commit a5fac1ef09
2 changed files with 5 additions and 2 deletions

View File

@@ -188,7 +188,10 @@ export const streamBedrock: StreamFunction<"bedrock-converse-stream", BedrockOpt
modelId: model.id,
messages: convertMessages(context, model, cacheRetention),
system: buildSystemPrompt(context.systemPrompt, model, cacheRetention),
inferenceConfig: { maxTokens: options.maxTokens, temperature: options.temperature },
inferenceConfig: {
...(options.maxTokens !== undefined && { maxTokens: options.maxTokens }),
...(options.temperature !== undefined && { temperature: options.temperature }),
},
toolConfig: convertToolConfig(context.tools, options.toolChoice),
additionalModelRequestFields: buildAdditionalModelRequestFields(model, options),
...(options.requestMetadata !== undefined && { requestMetadata: options.requestMetadata }),

View File

@@ -3,7 +3,7 @@ import type { Api, Model, SimpleStreamOptions, StreamOptions, ThinkingBudgets, T
export function buildBaseOptions(model: Model<Api>, options?: SimpleStreamOptions, apiKey?: string): StreamOptions {
return {
temperature: options?.temperature,
maxTokens: options?.maxTokens || Math.min(model.maxTokens, 32000),
maxTokens: options?.maxTokens ?? (model.maxTokens > 0 ? Math.min(model.maxTokens, 32000) : undefined),
signal: options?.signal,
apiKey: apiKey || options?.apiKey,
cacheRetention: options?.cacheRetention,