docs(coding-agent): document overflow normalization for custom providers
Adds a Context Overflow Errors subsection to custom-provider.md showing how a custom provider extension can rewrite unrecognized overflow errors via a message_end handler so pi's native compact-and-retry path triggers. Includes guard rails for provider scoping, idempotency, and avoiding rate-limit phrases.
This commit is contained in:
@@ -23,6 +23,7 @@ See these complete provider examples:
|
|||||||
- [Unregister Provider](#unregister-provider)
|
- [Unregister Provider](#unregister-provider)
|
||||||
- [OAuth Support](#oauth-support)
|
- [OAuth Support](#oauth-support)
|
||||||
- [Custom Streaming API](#custom-streaming-api)
|
- [Custom Streaming API](#custom-streaming-api)
|
||||||
|
- [Context Overflow Errors](#context-overflow-errors)
|
||||||
- [Testing Your Implementation](#testing-your-implementation)
|
- [Testing Your Implementation](#testing-your-implementation)
|
||||||
- [Config Reference](#config-reference)
|
- [Config Reference](#config-reference)
|
||||||
- [Model Definition Reference](#model-definition-reference)
|
- [Model Definition Reference](#model-definition-reference)
|
||||||
@@ -506,6 +507,60 @@ output.usage.totalTokens = output.usage.input + output.usage.output +
|
|||||||
calculateCost(model, output.usage);
|
calculateCost(model, output.usage);
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Context Overflow Errors
|
||||||
|
|
||||||
|
When a request exceeds the model's context window, pi can recover automatically by compacting the conversation and retrying. This recovery only kicks in if pi recognizes the failure as an overflow.
|
||||||
|
|
||||||
|
Detection runs on the finalized assistant message:
|
||||||
|
|
||||||
|
- `stopReason === "error"`
|
||||||
|
- `errorMessage` matches one of pi's known overflow patterns (see [`packages/ai/src/utils/overflow.ts`](https://github.com/earendil-works/pi-mono/blob/main/packages/ai/src/utils/overflow.ts))
|
||||||
|
|
||||||
|
If your provider returns overflow errors with a message pi does not recognize, normalize the error from the same extension that registers the provider. Use a `message_end` handler to rewrite the assistant message so its `errorMessage` starts with a phrase pi recognizes. The generic fallback `context_length_exceeded` is the safest choice.
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const MY_PROVIDER_OVERFLOW_PATTERN = /your provider's overflow phrase/i;
|
||||||
|
|
||||||
|
export default function (pi: ExtensionAPI) {
|
||||||
|
pi.registerProvider("my-provider", { /* ... */ });
|
||||||
|
|
||||||
|
pi.on("message_end", (event, ctx) => {
|
||||||
|
const message = event.message;
|
||||||
|
if (message.role !== "assistant") return;
|
||||||
|
if (message.stopReason !== "error") return;
|
||||||
|
if (
|
||||||
|
message.provider !== "my-provider" &&
|
||||||
|
ctx.model?.provider !== "my-provider"
|
||||||
|
)
|
||||||
|
return;
|
||||||
|
|
||||||
|
const errorMessage = message.errorMessage ?? "";
|
||||||
|
if (errorMessage.includes("context_length_exceeded")) return;
|
||||||
|
if (!MY_PROVIDER_OVERFLOW_PATTERN.test(errorMessage)) return;
|
||||||
|
|
||||||
|
return {
|
||||||
|
message: {
|
||||||
|
...message,
|
||||||
|
errorMessage: `context_length_exceeded: ${errorMessage}`,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`message_end` runs before pi tracks the assistant message for auto-compaction, so the rewritten `errorMessage` is what pi checks. With this in place, pi will:
|
||||||
|
|
||||||
|
1. Detect the overflow from `errorMessage`.
|
||||||
|
2. Drop the failed assistant message from live context.
|
||||||
|
3. Run compaction.
|
||||||
|
4. Retry the request once.
|
||||||
|
|
||||||
|
Guard the rewrite carefully:
|
||||||
|
|
||||||
|
- Scope it to your provider (`message.provider` and `ctx.model?.provider`) so unrelated errors from other providers are untouched.
|
||||||
|
- Match a provider-specific pattern, not pi's generic overflow patterns. Rewriting rate-limit or throttling errors (`rate limit`, `too many requests`) would falsely trigger compaction instead of pi's normal retry-with-backoff path.
|
||||||
|
- Skip when `errorMessage` already includes `context_length_exceeded` so the handler is idempotent.
|
||||||
|
|
||||||
### Registration
|
### Registration
|
||||||
|
|
||||||
Register your stream function:
|
Register your stream function:
|
||||||
|
|||||||
Reference in New Issue
Block a user