docs: clarify compat flags for openai-compatible local servers closes #2177
This commit is contained in:
@@ -729,6 +729,29 @@ const response = await stream(ollamaModel, context, {
|
||||
});
|
||||
```
|
||||
|
||||
Some OpenAI-compatible servers do not understand the `developer` role used for reasoning-capable models. For those providers, set `compat.supportsDeveloperRole` to `false` so the system prompt is sent as a `system` message instead. If the server also does not support `reasoning_effort`, set `compat.supportsReasoningEffort` to `false` too.
|
||||
|
||||
This commonly applies to Ollama, vLLM, SGLang, and similar OpenAI-compatible servers. You can set `compat` at the provider level or per model.
|
||||
|
||||
```typescript
|
||||
const ollamaReasoningModel: Model<'openai-completions'> = {
|
||||
id: 'gpt-oss:20b',
|
||||
name: 'GPT-OSS 20B (Ollama)',
|
||||
api: 'openai-completions',
|
||||
provider: 'ollama',
|
||||
baseUrl: 'http://localhost:11434/v1',
|
||||
reasoning: true,
|
||||
input: ['text'],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 131072,
|
||||
maxTokens: 32000,
|
||||
compat: {
|
||||
supportsDeveloperRole: false,
|
||||
supportsReasoningEffort: false,
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### OpenAI Compatibility Settings
|
||||
|
||||
The `openai-completions` API is implemented by many providers with minor differences. By default, the library auto-detects compatibility settings based on `baseUrl` for a small set of known OpenAI-compatible providers (Cerebras, xAI, Chutes, DeepSeek, zAi, OpenCode, etc.). For custom proxies or unknown endpoints, you can override these settings via the `compat` field. For `openai-responses` models, the compat field only supports Responses-specific flags.
|
||||
|
||||
Reference in New Issue
Block a user