62d9eefc2a
- Extract and display reasoning tokens from both Chat Completions and Responses APIs
- Add smart preflight detection to check reasoning support per model/API (cached per agent)
- Support both reasoning_text (o1/o3) and summary_text (gpt-5) formats
- Display reasoning tokens with ⚡ symbol in console and TUI renderers
- Only send reasoning parameters to models that support them
- Fix event type from "thinking" to "reasoning" for consistency
Note: Chat Completions API only returns reasoning token counts, not content (by design).
Only Responses API exposes actual thinking/reasoning events.
40 lines
3.1 KiB
Markdown
40 lines
3.1 KiB
Markdown
# Fix Missing Thinking Tokens for GPT-5 and Anthropic Models
|
|
**Status:** AwaitingCommit
|
|
**Agent PID:** 27674
|
|
|
|
## Original Todo
|
|
agent: we do not get thinking tokens for gpt-5. possibly also not for anthropic models?
|
|
|
|
## Description
|
|
The agent doesn't extract or report reasoning/thinking tokens from OpenAI's reasoning models (gpt-5, o1, o3) when using the Chat Completions API. While the codebase has full thinking token support for the Responses API, the Chat Completions API implementation is missing the extraction of `reasoning_tokens` from the `usage.completion_tokens_details` object. This means users don't see how many tokens were used for reasoning, which can be significant (thousands of tokens) for these models.
|
|
|
|
*Read [analysis.md](./analysis.md) in full for detailed codebase research and context*
|
|
|
|
## Implementation Plan
|
|
- [x] Extend AgentEvent token_usage type to include reasoningTokens field (packages/agent/src/agent.ts:16-23)
|
|
- [x] Update Chat Completions API token extraction to include reasoning tokens from usage.completion_tokens_details (packages/agent/src/agent.ts:210-220)
|
|
- [x] Update console renderer to display reasoning tokens in usage metrics (packages/agent/src/renderers/console-renderer.ts:117-121)
|
|
- [x] Update TUI renderer to display reasoning tokens in usage metrics (packages/agent/src/renderers/tui-renderer.ts:219-227)
|
|
- [x] Update JSON renderer to include reasoning tokens in output (packages/agent/src/renderers/json-renderer.ts:20)
|
|
- [x] User test: Run agent with gpt-4o-mini model (or other reasoning model) and verify reasoning token count appears in metrics display
|
|
- [x] Debug: Fix missing reasoningTokens field in JSON output even when value is 0
|
|
- [x] Debug: Investigate why o3 model doesn't report reasoning tokens in responses API
|
|
- [x] Fix: Parse reasoning summaries from gpt-5 models (summary_text vs reasoning_text)
|
|
- [x] Fix: Only send reasoning parameter for models that support it (o3, gpt-5, etc)
|
|
- [x] Fix: Better detection of reasoning support - preflight test instead of hardcoded model names
|
|
- [x] Fix: Add reasoning support detection for Chat Completions API
|
|
- [x] Fix: Add correct summary parameter value and increase max_output_tokens for preflight check
|
|
- [x] Investigate: Chat Completions API has reasoning tokens but no thinking events
|
|
|
|
## Notes
|
|
User reported that o3 model with responses API doesn't show reasoning tokens or thinking events.
|
|
Fixed by:
|
|
1. Adding reasoningTokens field to AgentEvent type
|
|
2. Extracting reasoning tokens from both Chat Completions and Responses APIs
|
|
3. Smart preflight detection of reasoning support for both APIs (cached per agent instance)
|
|
4. Only sending reasoning parameter for supported models
|
|
5. Parsing both reasoning_text (o1/o3) and summary_text (gpt-5) formats
|
|
6. Displaying reasoning tokens in console and TUI renderers with ⚡ symbol
|
|
7. Properly handling reasoning_effort for Chat Completions API
|
|
|
|
**Important finding**: Chat Completions API by design only returns reasoning token *counts* but not the actual thinking/reasoning content for o1 models. This is expected behavior - only the Responses API exposes thinking events. |