refactor(agent): run harness loop directly
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# AgentHarness lifecycle
|
||||
|
||||
`AgentHarness` is the orchestration layer above the low-level `Agent`. It owns session persistence, runtime configuration, resource resolution, operation locking, and extension-facing mutation semantics.
|
||||
`AgentHarness` is the orchestration layer above the low-level agent loop. It owns session persistence, runtime configuration, resource resolution, operation locking, and extension-facing mutation semantics.
|
||||
|
||||
This document describes the current direction and implemented behavior. Some extension/session-facade details are planned and called out explicitly.
|
||||
|
||||
@@ -15,6 +15,7 @@ The intended rule is:
|
||||
- runtime config setters update future snapshots without mutating the current provider request
|
||||
- session writes made while busy are durably queued and flushed in deterministic order
|
||||
- getters return latest harness config, not in-flight snapshots
|
||||
- listeners/hooks currently receive no facade; if they close over the raw harness and call settlement APIs such as `waitForIdle()` during the active run, they can deadlock. A future facade should expose `runWhenIdle()` instead.
|
||||
|
||||
A final lifecycle hardening pass should prove these guarantees with a broad listener/hook reentrancy test suite.
|
||||
|
||||
@@ -123,8 +124,8 @@ Phase/settlement semantics are still provisional and need a full lifecycle pass.
|
||||
|
||||
Queue modes are live, not turn-snapshotted:
|
||||
|
||||
- `steeringMode`
|
||||
- `followUpMode`
|
||||
- `getSteeringMode()` / `setSteeringMode()`
|
||||
- `getFollowUpMode()` / `setFollowUpMode()`
|
||||
|
||||
Changing a queue mode during a run affects the next queue drain. Queue drains happen at safe points.
|
||||
|
||||
@@ -151,17 +152,17 @@ If the system-prompt callback throws while starting `prompt`, `skill`, or `promp
|
||||
|
||||
## Hooks and events
|
||||
|
||||
Current hooks receive only the event payload. There is no extension context object yet.
|
||||
Current hooks and listeners receive only the event payload. There is no extension context object yet.
|
||||
|
||||
Event payloads describe what is happening. Harness getters describe latest config for future snapshots.
|
||||
|
||||
The split between harness-specific events (`AgentHarnessOwnEvent`) and the union of low-level plus harness events (`AgentHarnessEvent`) is provisional but useful for distinguishing hookable harness events from public subscription events.
|
||||
|
||||
A future extension context may expose the harness and a queued-write session facade.
|
||||
A future extension context should expose a harness facade plus a queued-write session facade. The facade must not expose APIs that can deadlock the current event dispatch. In particular, listeners/hooks should not call `waitForIdle()` for the active run; expose a `runWhenIdle(() => Promise<void>)` scheduling API instead. This is future extension-context work; current listeners/hooks receive only payloads and no safe harness facade.
|
||||
|
||||
## Planned session facade
|
||||
|
||||
Extensions should eventually interact with a harness-scoped session facade rather than the raw session.
|
||||
Extensions should eventually interact with a harness-scoped `HarnessSession` facade rather than the raw session. The facade should wrap the internal session and enforce harness pending-write ordering semantics. Once this exists, hooks and event listeners can receive a context that exposes the full `AgentHarness` plus the session facade without giving direct access to unordered raw session writes.
|
||||
|
||||
Planned read semantics:
|
||||
|
||||
@@ -183,7 +184,9 @@ Agent-emitted messages are persisted on `message_end` to preserve transcript ord
|
||||
|
||||
## Abort
|
||||
|
||||
Abort is allowed during a turn. It aborts the low-level run and clears low-level steering/follow-up queues.
|
||||
Abort is allowed during a turn. It aborts the low-level run and clears steering/follow-up queues.
|
||||
|
||||
Abort does not clear `nextTurn` messages. Messages queued with `nextTurn()` survive abort and are inserted before the user message on the next user-initiated turn.
|
||||
|
||||
Abort does not discard pending session writes. Pending writes flush at the next save point if reached, at `agent_end`, or in operation failure cleanup.
|
||||
|
||||
@@ -205,7 +208,7 @@ Harness tests should stay focused by area instead of growing one large catch-all
|
||||
|
||||
Current structure:
|
||||
|
||||
- `packages/agent/test/harness/agent-harness.test.ts`: basic construction/API smoke tests.
|
||||
- `packages/agent/test/harness/agent-harness.test.ts`: core lifecycle and public API behavior.
|
||||
- `packages/agent/test/harness/agent-harness-stream.test.ts`: stream options and provider hook semantics.
|
||||
|
||||
Preferred future structure:
|
||||
@@ -216,27 +219,40 @@ Preferred future structure:
|
||||
|
||||
Use the `pi-ai` faux provider (`registerFauxProvider`, `fauxAssistantMessage`) for deterministic harness/provider tests. Faux response factories can inspect `StreamOptions`, invoke `options.onPayload`, and return scripted assistant messages without real provider APIs or network access.
|
||||
|
||||
Harness coverage is configured separately from the default package test run:
|
||||
|
||||
```bash
|
||||
npm run test:harness
|
||||
npm run coverage:harness
|
||||
```
|
||||
|
||||
`coverage:harness` runs `test/harness/**/*.test.ts` and reports coverage for `src/harness/**/*.ts` plus the non-harness runtime files it directly exercises (`src/agent.ts` and `src/agent-loop.ts`) into `coverage/harness`. Type-only dependencies such as `src/types.ts` are not included because they have no meaningful runtime coverage.
|
||||
|
||||
## Implementation todo
|
||||
|
||||
This list tracks the remaining work before treating `AgentHarness` as migration-ready.
|
||||
|
||||
### 1. Remove `Agent` dependency from `AgentHarness`
|
||||
|
||||
New top priority.
|
||||
Implemented.
|
||||
|
||||
`AgentHarness` should likely call `agentLoop` / `agentLoopContinue` directly instead of owning an internal `Agent` instance. The harness already owns session persistence, runtime config snapshots, queues, provider stream configuration, hooks/events, phase semantics, and abort semantics. Keeping `Agent` in the middle creates duplicated state and adapter seams.
|
||||
`AgentHarness` now calls `runAgentLoop()` directly instead of owning an internal `Agent` instance. The harness owns active run/abort-controller lifecycle, queue draining, provider stream configuration, event reduction, session persistence, pending write flushing, and save-point snapshot refresh.
|
||||
|
||||
Still needed:
|
||||
Implemented validation:
|
||||
|
||||
- Replace internal `new Agent(...)` with direct low-level loop calls.
|
||||
- Move active run/abort-controller lifecycle into `AgentHarness`.
|
||||
- Move queue draining into `AgentHarness` only, removing duplicated low-level `Agent` queues.
|
||||
- Reduce low-level `AgentEvent` state directly in the harness where needed.
|
||||
- Preserve current public behavior for `prompt`, `skill`, `promptFromTemplate`, `steer`, `followUp`, `nextTurn`, `abort`, and `waitForIdle`.
|
||||
- Preserve provider hook behavior implemented by the harness stream wrapper.
|
||||
- Preserve save-point snapshot refresh semantics without side-effecting through `Agent.prepareNextTurn`.
|
||||
- Decide whether `AgentHarness.agent` remains temporarily for compatibility or is removed before migration.
|
||||
- Add tests covering parity with the current harness behavior before and after the refactor.
|
||||
- prompt construction and public runtime config getters/mutators
|
||||
- steering queue draining and `queue_update` emission
|
||||
- follow-up queue draining and `queue_update` emission
|
||||
- `before_agent_start` message ordering and persistence
|
||||
- abort clearing steer/follow-up queues while preserving `nextTurn` messages
|
||||
- thrown hook failure cleanup with persisted assistant error messages and settlement
|
||||
- save-point refresh for model, thinking level, resources, system prompt, and active tools
|
||||
- pending listener session write ordering after agent-emitted messages
|
||||
- external `waitForIdle()` waiting for awaited listeners and run settlement
|
||||
- `tool_call` and `tool_result` hook behavior through the direct loop
|
||||
- provider stream wrapper behavior in `agent-harness-stream.test.ts`
|
||||
|
||||
Remaining lifecycle hardening beyond this refactor is tracked in the final lifecycle hardening suite.
|
||||
|
||||
### 2. Finish curated provider/stream configuration
|
||||
|
||||
@@ -294,8 +310,10 @@ Current cleanup already done:
|
||||
|
||||
Still needed:
|
||||
|
||||
- Define extension context shape.
|
||||
- Likely expose a harness facade plus a session facade rather than raw internals.
|
||||
- Define extension/listener context shape.
|
||||
- Expose a harness facade plus a session facade rather than raw internals.
|
||||
- The harness facade should expose safe runtime APIs and `runWhenIdle(() => Promise<void>)`; it should not expose active-run `waitForIdle()` to listeners/hooks.
|
||||
- The session facade should wrap the internal session and participate in pending session write queue semantics so writes remain ordered with agent-emitted messages.
|
||||
- Decide which public harness APIs are allowed from each hook/event.
|
||||
- Decide whether hooks can mutate turn snapshots directly or only through explicit hook results/public APIs.
|
||||
- Clarify event payload semantics versus harness getter semantics.
|
||||
@@ -315,9 +333,9 @@ Implemented so far:
|
||||
- `setActiveTools(toolNames)`
|
||||
- invalid active tool names currently throw; convert to result errors
|
||||
- generic common app tool shape via `AgentHarness<TSkill, TPromptTemplate, TTool>`
|
||||
- `QueueMode` exported from `Agent`
|
||||
- `QueueMode` exported from core types
|
||||
- `AgentHarnessOptions.steeringMode` / `followUpMode`
|
||||
- live `steeringMode` / `followUpMode` getters/setters
|
||||
- live `getSteeringMode()` / `setSteeringMode()` and `getFollowUpMode()` / `setFollowUpMode()` methods
|
||||
- queue modes are immediate/live, matching coding-agent behavior
|
||||
|
||||
Still needed:
|
||||
@@ -355,7 +373,7 @@ Still needed:
|
||||
- Implement retry handling.
|
||||
- Ensure structural operations use consistent `try/finally` phase cleanup.
|
||||
- Verify `before_agent_start` hook semantics against coding-agent:
|
||||
- current behavior prepends returned messages.
|
||||
- current behavior appends returned messages after the user/next-turn prompt messages.
|
||||
- decide whether replacement, prepend, append, or transform semantics are correct.
|
||||
- Decide if `before_agent_start` needs more turn info such as tools/tool snippets.
|
||||
- Document or change timing for model/thinking/stream-option events that may fire before queued session entries persist while busy.
|
||||
@@ -373,6 +391,7 @@ Implemented so far:
|
||||
- Updated skill and prompt-template loaders to consume `ExecutionEnv` results.
|
||||
- Updated shell output capture to return a result and use `ExecutionEnv` instead of Node APIs directly, including full-output spill via `appendFile()`.
|
||||
- Removed `NodeExecutionEnv` from the browser-safe `execution-env.ts` re-export; Node-specific callers import from `harness/env/nodejs.js`.
|
||||
- Replaced `Buffer` usage in generic truncation utilities with runtime-neutral UTF-8 handling.
|
||||
- Expanded `NodeExecutionEnv` tests for file operations, exec errors, aborts, callbacks, timeouts, and shell-output full-output spill.
|
||||
|
||||
Still needed:
|
||||
@@ -382,7 +401,7 @@ Still needed:
|
||||
- Convert structural `AgentHarness` operations to typed result returns for busy, missing-resource, auth, compaction, and branch-summary failures.
|
||||
- Convert compaction helpers to typed result returns.
|
||||
- Keep Node-specific APIs isolated under `src/harness/env/nodejs.ts` and Node-backed storage/session implementations, or move those implementations behind explicit Node-only entry points.
|
||||
- Replace Node globals in generic harness utilities, especially `Buffer` usage in truncation utilities, with runtime-neutral implementations.
|
||||
- Audit remaining generic harness utilities for Node globals as new APIs are added.
|
||||
- Audit package exports so browser/generic-JS imports do not pull Node-only modules such as `NodeExecutionEnv` or JSONL storage.
|
||||
- Keep expanding `ExecutionEnv` and shell-output contract tests as the API evolves, especially for non-Node implementations.
|
||||
- Add tests proving harness APIs return `ok: false` instead of throwing for expected failure paths.
|
||||
|
||||
@@ -14,6 +14,8 @@
|
||||
"build": "tsgo -p tsconfig.build.json",
|
||||
"dev": "tsgo -p tsconfig.build.json --watch --preserveWatchOutput",
|
||||
"test": "vitest --run",
|
||||
"test:harness": "vitest --run --config vitest.harness.config.ts",
|
||||
"coverage:harness": "vitest --run --config vitest.harness.config.ts --coverage",
|
||||
"prepublishOnly": "npm run clean && npm run build"
|
||||
},
|
||||
"dependencies": {
|
||||
@@ -41,6 +43,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.3.0",
|
||||
"@vitest/coverage-v8": "^3.2.4",
|
||||
"typescript": "^5.7.3",
|
||||
"vitest": "^3.2.4"
|
||||
}
|
||||
|
||||
@@ -21,10 +21,13 @@ import type {
|
||||
AgentTool,
|
||||
BeforeToolCallContext,
|
||||
BeforeToolCallResult,
|
||||
QueueMode,
|
||||
StreamFn,
|
||||
ToolExecutionMode,
|
||||
} from "./types.js";
|
||||
|
||||
export type { QueueMode } from "./types.js";
|
||||
|
||||
function defaultConvertToLlm(messages: AgentMessage[]): Message[] {
|
||||
return messages.filter(
|
||||
(message) => message.role === "user" || message.role === "assistant" || message.role === "toolResult",
|
||||
@@ -53,8 +56,6 @@ const DEFAULT_MODEL = {
|
||||
maxTokens: 0,
|
||||
} satisfies Model<any>;
|
||||
|
||||
export type QueueMode = "all" | "one-at-a-time";
|
||||
|
||||
type MutableAgentState = Omit<AgentState, "isStreaming" | "streamingMessage" | "pendingToolCalls" | "errorMessage"> & {
|
||||
isStreaming: boolean;
|
||||
streamingMessage?: AgentMessage;
|
||||
|
||||
@@ -5,10 +5,20 @@ import {
|
||||
streamSimple,
|
||||
type UserMessage,
|
||||
} from "@earendil-works/pi-ai";
|
||||
import { Agent, type QueueMode } from "../agent.js";
|
||||
import type { AgentEvent, AgentMessage, AgentTool, ThinkingLevel } from "../types.js";
|
||||
import { runAgentLoop } from "../agent-loop.js";
|
||||
import type {
|
||||
AgentContext,
|
||||
AgentEvent,
|
||||
AgentLoopConfig,
|
||||
AgentMessage,
|
||||
AgentTool,
|
||||
QueueMode,
|
||||
StreamFn,
|
||||
ThinkingLevel,
|
||||
} from "../types.js";
|
||||
import { collectEntriesForBranchSummary, generateBranchSummary } from "./compaction/branch-summarization.js";
|
||||
import { compact, DEFAULT_COMPACTION_SETTINGS, prepareCompaction } from "./compaction/compaction.js";
|
||||
import { convertToLlm } from "./messages.js";
|
||||
import { formatPromptTemplateInvocation } from "./prompt-templates.js";
|
||||
import { formatSkillInvocation } from "./skills.js";
|
||||
import type {
|
||||
@@ -35,6 +45,27 @@ function createUserMessage(text: string, images?: ImageContent[]): UserMessage {
|
||||
return { role: "user", content, timestamp: Date.now() };
|
||||
}
|
||||
|
||||
function createFailureMessage(model: Model<any>, error: unknown, aborted: boolean): AssistantMessage {
|
||||
return {
|
||||
role: "assistant",
|
||||
content: [{ type: "text", text: "" }],
|
||||
api: model.api,
|
||||
provider: model.provider,
|
||||
model: model.id,
|
||||
stopReason: aborted ? "aborted" : "error",
|
||||
errorMessage: error instanceof Error ? error.message : String(error),
|
||||
timestamp: Date.now(),
|
||||
usage: {
|
||||
input: 0,
|
||||
output: 0,
|
||||
cacheRead: 0,
|
||||
cacheWrite: 0,
|
||||
totalTokens: 0,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function cloneStreamOptions(streamOptions?: AgentHarnessStreamOptions): AgentHarnessStreamOptions {
|
||||
return {
|
||||
...streamOptions,
|
||||
@@ -54,10 +85,6 @@ function mergeHeaders(...headers: Array<Record<string, string> | undefined>): Re
|
||||
return hasHeaders ? merged : undefined;
|
||||
}
|
||||
|
||||
function hasOwn(object: object, key: PropertyKey): boolean {
|
||||
return Object.hasOwn(object, key);
|
||||
}
|
||||
|
||||
function applyStreamOptionsPatch(
|
||||
base: AgentHarnessStreamOptions,
|
||||
patch?: AgentHarnessStreamOptionsPatch,
|
||||
@@ -65,13 +92,13 @@ function applyStreamOptionsPatch(
|
||||
const result = cloneStreamOptions(base);
|
||||
if (!patch) return result;
|
||||
|
||||
if (hasOwn(patch, "transport")) result.transport = patch.transport;
|
||||
if (hasOwn(patch, "timeoutMs")) result.timeoutMs = patch.timeoutMs;
|
||||
if (hasOwn(patch, "maxRetries")) result.maxRetries = patch.maxRetries;
|
||||
if (hasOwn(patch, "maxRetryDelayMs")) result.maxRetryDelayMs = patch.maxRetryDelayMs;
|
||||
if (hasOwn(patch, "cacheRetention")) result.cacheRetention = patch.cacheRetention;
|
||||
if (Object.hasOwn(patch, "transport")) result.transport = patch.transport;
|
||||
if (Object.hasOwn(patch, "timeoutMs")) result.timeoutMs = patch.timeoutMs;
|
||||
if (Object.hasOwn(patch, "maxRetries")) result.maxRetries = patch.maxRetries;
|
||||
if (Object.hasOwn(patch, "maxRetryDelayMs")) result.maxRetryDelayMs = patch.maxRetryDelayMs;
|
||||
if (Object.hasOwn(patch, "cacheRetention")) result.cacheRetention = patch.cacheRetention;
|
||||
|
||||
if (hasOwn(patch, "headers")) {
|
||||
if (Object.hasOwn(patch, "headers")) {
|
||||
if (patch.headers === undefined) {
|
||||
result.headers = undefined;
|
||||
} else {
|
||||
@@ -84,7 +111,7 @@ function applyStreamOptionsPatch(
|
||||
}
|
||||
}
|
||||
|
||||
if (hasOwn(patch, "metadata")) {
|
||||
if (Object.hasOwn(patch, "metadata")) {
|
||||
if (patch.metadata === undefined) {
|
||||
result.metadata = undefined;
|
||||
} else {
|
||||
@@ -100,6 +127,10 @@ function applyStreamOptionsPatch(
|
||||
return result;
|
||||
}
|
||||
|
||||
const SUBSCRIBER_EVENT_TYPE = "*";
|
||||
|
||||
type AgentHarnessHandler = (event: any, signal?: AbortSignal) => Promise<any> | any;
|
||||
|
||||
interface AgentHarnessTurnState<
|
||||
TSkill extends Skill = Skill,
|
||||
TPromptTemplate extends PromptTemplate = PromptTemplate,
|
||||
@@ -121,72 +152,28 @@ export class AgentHarness<
|
||||
TPromptTemplate extends PromptTemplate = PromptTemplate,
|
||||
TTool extends AgentTool = AgentTool,
|
||||
> {
|
||||
readonly agent: Agent;
|
||||
readonly env: ExecutionEnv;
|
||||
private session: Session;
|
||||
private phase: AgentHarnessPhase = "idle";
|
||||
private runAbortController?: AbortController;
|
||||
private runPromise?: Promise<void>;
|
||||
private pendingSessionWrites: PendingSessionWrite[] = [];
|
||||
private model: Model<any>;
|
||||
private thinkingLevel: ThinkingLevel;
|
||||
private activeToolNames: string[];
|
||||
private nextTurnQueue: AgentMessage[] = [];
|
||||
private phase: AgentHarnessPhase = "idle";
|
||||
private steerQueue: UserMessage[] = [];
|
||||
private followUpQueue: UserMessage[] = [];
|
||||
private pendingSessionWrites: PendingSessionWrite[] = [];
|
||||
private resources: AgentHarnessResources<TSkill, TPromptTemplate>;
|
||||
private streamOptions: AgentHarnessStreamOptions;
|
||||
private appliedStreamOptions: AgentHarnessStreamOptions = {};
|
||||
private appliedSessionId?: string;
|
||||
private systemPrompt: AgentHarnessOptions<TSkill, TPromptTemplate, TTool>["systemPrompt"];
|
||||
private streamOptions: AgentHarnessStreamOptions;
|
||||
private getApiKeyAndHeaders?: AgentHarnessOptions["getApiKeyAndHeaders"];
|
||||
private resources: AgentHarnessResources<TSkill, TPromptTemplate>;
|
||||
private tools = new Map<string, TTool>();
|
||||
private listeners = new Set<
|
||||
(event: AgentHarnessEvent<TSkill, TPromptTemplate>, signal?: AbortSignal) => Promise<void> | void
|
||||
>();
|
||||
private hooks = new Map<keyof AgentHarnessEventResultMap, Set<(event: any) => Promise<any> | any>>();
|
||||
private activeToolNames: string[];
|
||||
private steerQueue: UserMessage[] = [];
|
||||
private steeringQueueMode: QueueMode;
|
||||
private followUpQueue: UserMessage[] = [];
|
||||
private followUpQueueMode: QueueMode;
|
||||
private nextTurnQueue: AgentMessage[] = [];
|
||||
private handlers = new Map<string, Set<AgentHarnessHandler>>();
|
||||
|
||||
constructor(options: AgentHarnessOptions<TSkill, TPromptTemplate, TTool>) {
|
||||
this.agent = new Agent({
|
||||
initialState: {
|
||||
model: options.model,
|
||||
thinkingLevel: options.thinkingLevel,
|
||||
tools: options.tools ?? [],
|
||||
},
|
||||
streamFn: async (model, context, streamOptions) => {
|
||||
const auth = await this.getApiKeyAndHeaders?.(model);
|
||||
const snapshotOptions: AgentHarnessStreamOptions = {
|
||||
...this.appliedStreamOptions,
|
||||
headers: mergeHeaders(this.appliedStreamOptions.headers, auth?.headers),
|
||||
};
|
||||
const requestOptions = await this.emitBeforeProviderRequest(
|
||||
model,
|
||||
this.appliedSessionId ?? "",
|
||||
snapshotOptions,
|
||||
);
|
||||
return streamSimple(model, context, {
|
||||
cacheRetention: requestOptions.cacheRetention,
|
||||
headers: requestOptions.headers,
|
||||
maxRetries: requestOptions.maxRetries,
|
||||
maxRetryDelayMs: requestOptions.maxRetryDelayMs,
|
||||
metadata: requestOptions.metadata,
|
||||
onPayload: async (payload) => await this.emitBeforeProviderPayload(model, payload),
|
||||
onResponse: async (response) => {
|
||||
const headers = { ...(response.headers as Record<string, string>) };
|
||||
await this.emitOwn(
|
||||
{ type: "after_provider_response", status: response.status, headers },
|
||||
this.agent.signal,
|
||||
);
|
||||
},
|
||||
reasoning: streamOptions?.reasoning,
|
||||
signal: streamOptions?.signal,
|
||||
sessionId: this.appliedSessionId,
|
||||
timeoutMs: requestOptions.timeoutMs,
|
||||
transport: requestOptions.transport,
|
||||
apiKey: auth?.apiKey,
|
||||
});
|
||||
},
|
||||
steeringMode: options.steeringMode,
|
||||
followUpMode: options.followUpMode,
|
||||
});
|
||||
this.env = options.env;
|
||||
this.session = options.session;
|
||||
this.resources = options.resources ?? {};
|
||||
@@ -197,64 +184,24 @@ export class AgentHarness<
|
||||
this.tools.set(tool.name, tool);
|
||||
}
|
||||
this.model = options.model;
|
||||
this.thinkingLevel = options.thinkingLevel ?? this.agent.state.thinkingLevel;
|
||||
this.thinkingLevel = options.thinkingLevel ?? "off";
|
||||
this.activeToolNames = options.activeToolNames ?? (options.tools ?? []).map((tool) => tool.name);
|
||||
this.agent.state.model = this.model;
|
||||
this.agent.state.thinkingLevel = this.thinkingLevel;
|
||||
this.agent.transformContext = async (messages) => {
|
||||
const result = await this.emitHook({ type: "context", messages: [...messages] });
|
||||
return result?.messages ?? messages;
|
||||
};
|
||||
this.agent.beforeToolCall = async ({ toolCall, args }) => {
|
||||
const result = await this.emitHook({
|
||||
type: "tool_call",
|
||||
toolCallId: toolCall.id,
|
||||
toolName: toolCall.name,
|
||||
input: args as Record<string, unknown>,
|
||||
});
|
||||
return result ? { block: result.block, reason: result.reason } : undefined;
|
||||
};
|
||||
this.agent.afterToolCall = async ({ toolCall, args, result, isError }) => {
|
||||
const patch = await this.emitHook({
|
||||
type: "tool_result",
|
||||
toolCallId: toolCall.id,
|
||||
toolName: toolCall.name,
|
||||
input: args as Record<string, unknown>,
|
||||
content: result.content,
|
||||
details: result.details,
|
||||
isError,
|
||||
});
|
||||
return patch
|
||||
? { content: patch.content, details: patch.details, isError: patch.isError, terminate: patch.terminate }
|
||||
: undefined;
|
||||
};
|
||||
this.agent.prepareNextTurn = async () => {
|
||||
await this.flushPendingSessionWrites();
|
||||
const turnState = await this.createTurnState();
|
||||
this.applyTurnState(turnState);
|
||||
return {
|
||||
context: {
|
||||
systemPrompt: turnState.systemPrompt,
|
||||
messages: turnState.messages.slice(),
|
||||
tools: turnState.activeTools.slice(),
|
||||
},
|
||||
model: turnState.model,
|
||||
thinkingLevel: turnState.thinkingLevel,
|
||||
};
|
||||
};
|
||||
this.agent.subscribe(async (event, signal) => {
|
||||
await this.handleAgentEvent(event, signal);
|
||||
});
|
||||
this.steeringQueueMode = options.steeringMode ?? "one-at-a-time";
|
||||
this.followUpQueueMode = options.followUpMode ?? "one-at-a-time";
|
||||
}
|
||||
|
||||
private getHandlers(type: string): Set<AgentHarnessHandler> | undefined {
|
||||
return this.handlers.get(type);
|
||||
}
|
||||
|
||||
private async emitOwn(event: AgentHarnessOwnEvent<TSkill, TPromptTemplate>, signal?: AbortSignal): Promise<void> {
|
||||
for (const listener of this.listeners) {
|
||||
for (const listener of this.getHandlers(SUBSCRIBER_EVENT_TYPE) ?? []) {
|
||||
await listener(event, signal);
|
||||
}
|
||||
}
|
||||
|
||||
private async emitAny(event: AgentHarnessEvent<TSkill, TPromptTemplate>, signal?: AbortSignal): Promise<void> {
|
||||
for (const listener of this.listeners) {
|
||||
for (const listener of this.getHandlers(SUBSCRIBER_EVENT_TYPE) ?? []) {
|
||||
await listener(event, signal);
|
||||
}
|
||||
}
|
||||
@@ -262,7 +209,7 @@ export class AgentHarness<
|
||||
private async emitHook<TType extends keyof AgentHarnessEventResultMap>(
|
||||
event: Extract<AgentHarnessOwnEvent, { type: TType }>,
|
||||
): Promise<AgentHarnessEventResultMap[TType] | undefined> {
|
||||
const handlers = this.hooks.get(event.type as TType);
|
||||
const handlers = this.getHandlers(event.type as TType);
|
||||
if (!handlers || handlers.size === 0) return undefined;
|
||||
let lastResult: AgentHarnessEventResultMap[TType] | undefined;
|
||||
for (const handler of handlers) {
|
||||
@@ -279,7 +226,7 @@ export class AgentHarness<
|
||||
sessionId: string,
|
||||
streamOptions: AgentHarnessStreamOptions,
|
||||
): Promise<AgentHarnessStreamOptions> {
|
||||
const handlers = this.hooks.get("before_provider_request");
|
||||
const handlers = this.getHandlers("before_provider_request");
|
||||
let current = cloneStreamOptions(streamOptions);
|
||||
if (!handlers || handlers.size === 0) return current;
|
||||
for (const handler of handlers) {
|
||||
@@ -297,7 +244,7 @@ export class AgentHarness<
|
||||
}
|
||||
|
||||
private async emitBeforeProviderPayload(model: Model<any>, payload: unknown): Promise<unknown> {
|
||||
const handlers = this.hooks.get("before_provider_payload");
|
||||
const handlers = this.getHandlers("before_provider_payload");
|
||||
let current = payload;
|
||||
if (!handlers || handlers.size === 0) return current;
|
||||
for (const handler of handlers) {
|
||||
@@ -318,6 +265,17 @@ export class AgentHarness<
|
||||
});
|
||||
}
|
||||
|
||||
private startRunPromise(): () => void {
|
||||
let finish = () => {};
|
||||
this.runPromise = new Promise<void>((resolve) => {
|
||||
finish = resolve;
|
||||
});
|
||||
return () => {
|
||||
this.runPromise = undefined;
|
||||
finish();
|
||||
};
|
||||
}
|
||||
|
||||
private async createTurnState(): Promise<AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>> {
|
||||
const context = await this.session.buildContext();
|
||||
const resources = this.getResources();
|
||||
@@ -352,14 +310,105 @@ export class AgentHarness<
|
||||
};
|
||||
}
|
||||
|
||||
private applyTurnState(turnState: AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>): void {
|
||||
this.agent.state.messages = turnState.messages;
|
||||
this.appliedStreamOptions = cloneStreamOptions(turnState.streamOptions);
|
||||
this.appliedSessionId = turnState.sessionId;
|
||||
this.agent.state.systemPrompt = turnState.systemPrompt;
|
||||
this.agent.state.model = turnState.model;
|
||||
this.agent.state.thinkingLevel = turnState.thinkingLevel;
|
||||
this.agent.state.tools = turnState.activeTools;
|
||||
private createContext(
|
||||
turnState: AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>,
|
||||
systemPrompt?: string,
|
||||
): AgentContext {
|
||||
return {
|
||||
systemPrompt: systemPrompt ?? turnState.systemPrompt,
|
||||
messages: turnState.messages.slice(),
|
||||
tools: turnState.activeTools.slice(),
|
||||
};
|
||||
}
|
||||
|
||||
private createStreamFn(getTurnState: () => AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>): StreamFn {
|
||||
return async (model, context, streamOptions) => {
|
||||
const turnState = getTurnState();
|
||||
const auth = await this.getApiKeyAndHeaders?.(model);
|
||||
const snapshotOptions: AgentHarnessStreamOptions = {
|
||||
...turnState.streamOptions,
|
||||
headers: mergeHeaders(turnState.streamOptions.headers, auth?.headers),
|
||||
};
|
||||
const requestOptions = await this.emitBeforeProviderRequest(model, turnState.sessionId, snapshotOptions);
|
||||
return streamSimple(model, context, {
|
||||
cacheRetention: requestOptions.cacheRetention,
|
||||
headers: requestOptions.headers,
|
||||
maxRetries: requestOptions.maxRetries,
|
||||
maxRetryDelayMs: requestOptions.maxRetryDelayMs,
|
||||
metadata: requestOptions.metadata,
|
||||
onPayload: async (payload) => await this.emitBeforeProviderPayload(model, payload),
|
||||
onResponse: async (response) => {
|
||||
const headers = { ...(response.headers as Record<string, string>) };
|
||||
await this.emitOwn(
|
||||
{ type: "after_provider_response", status: response.status, headers },
|
||||
streamOptions?.signal,
|
||||
);
|
||||
},
|
||||
reasoning: streamOptions?.reasoning,
|
||||
signal: streamOptions?.signal,
|
||||
sessionId: turnState.sessionId,
|
||||
timeoutMs: requestOptions.timeoutMs,
|
||||
transport: requestOptions.transport,
|
||||
apiKey: auth?.apiKey,
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
private async drainQueuedMessages(queue: AgentMessage[], mode: QueueMode): Promise<AgentMessage[]> {
|
||||
const messages = mode === "all" ? queue.splice(0) : queue.splice(0, 1);
|
||||
if (messages.length > 0) await this.emitQueueUpdate();
|
||||
return messages;
|
||||
}
|
||||
|
||||
private createLoopConfig(
|
||||
getTurnState: () => AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>,
|
||||
setTurnState: (turnState: AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>) => void,
|
||||
): AgentLoopConfig {
|
||||
const turnState = getTurnState();
|
||||
return {
|
||||
model: turnState.model,
|
||||
reasoning: turnState.thinkingLevel === "off" ? undefined : turnState.thinkingLevel,
|
||||
convertToLlm,
|
||||
transformContext: async (messages) => {
|
||||
const result = await this.emitHook({ type: "context", messages: [...messages] });
|
||||
return result?.messages ?? messages;
|
||||
},
|
||||
beforeToolCall: async ({ toolCall, args }) => {
|
||||
const result = await this.emitHook({
|
||||
type: "tool_call",
|
||||
toolCallId: toolCall.id,
|
||||
toolName: toolCall.name,
|
||||
input: args as Record<string, unknown>,
|
||||
});
|
||||
return result ? { block: result.block, reason: result.reason } : undefined;
|
||||
},
|
||||
afterToolCall: async ({ toolCall, args, result, isError }) => {
|
||||
const patch = await this.emitHook({
|
||||
type: "tool_result",
|
||||
toolCallId: toolCall.id,
|
||||
toolName: toolCall.name,
|
||||
input: args as Record<string, unknown>,
|
||||
content: result.content,
|
||||
details: result.details,
|
||||
isError,
|
||||
});
|
||||
return patch
|
||||
? { content: patch.content, details: patch.details, isError: patch.isError, terminate: patch.terminate }
|
||||
: undefined;
|
||||
},
|
||||
prepareNextTurn: async () => {
|
||||
await this.flushPendingSessionWrites();
|
||||
const nextTurnState = await this.createTurnState();
|
||||
setTurnState(nextTurnState);
|
||||
return {
|
||||
context: this.createContext(nextTurnState),
|
||||
model: nextTurnState.model,
|
||||
thinkingLevel: nextTurnState.thinkingLevel,
|
||||
};
|
||||
},
|
||||
getSteeringMessages: async () => this.drainQueuedMessages(this.steerQueue, this.steeringQueueMode),
|
||||
getFollowUpMessages: async () => this.drainQueuedMessages(this.followUpQueue, this.followUpQueueMode),
|
||||
};
|
||||
}
|
||||
|
||||
private validateToolNames(toolNames: string[]): void {
|
||||
@@ -391,19 +440,6 @@ export class AgentHarness<
|
||||
|
||||
private async handleAgentEvent(event: AgentEvent, signal?: AbortSignal): Promise<void> {
|
||||
await this.emitAny(event, signal);
|
||||
if (event.type === "message_start" && event.message.role === "user") {
|
||||
const steerIndex = this.steerQueue.indexOf(event.message);
|
||||
if (steerIndex !== -1) {
|
||||
this.steerQueue.splice(steerIndex, 1);
|
||||
await this.emitQueueUpdate();
|
||||
} else {
|
||||
const followUpIndex = this.followUpQueue.indexOf(event.message);
|
||||
if (followUpIndex !== -1) {
|
||||
this.followUpQueue.splice(followUpIndex, 1);
|
||||
await this.emitQueueUpdate();
|
||||
}
|
||||
}
|
||||
}
|
||||
if (event.type === "message_end") {
|
||||
await this.session.appendMessage(event.message);
|
||||
}
|
||||
@@ -422,13 +458,26 @@ export class AgentHarness<
|
||||
}
|
||||
}
|
||||
|
||||
private async emitRunFailure(
|
||||
model: Model<any>,
|
||||
error: unknown,
|
||||
aborted: boolean,
|
||||
signal: AbortSignal,
|
||||
): Promise<AgentMessage[]> {
|
||||
const failureMessage = createFailureMessage(model, error, aborted);
|
||||
await this.handleAgentEvent({ type: "message_start", message: failureMessage }, signal);
|
||||
await this.handleAgentEvent({ type: "message_end", message: failureMessage }, signal);
|
||||
await this.handleAgentEvent({ type: "turn_end", message: failureMessage, toolResults: [] }, signal);
|
||||
await this.handleAgentEvent({ type: "agent_end", messages: [failureMessage] }, signal);
|
||||
return [failureMessage];
|
||||
}
|
||||
|
||||
private async executeTurn(
|
||||
turnState: AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>,
|
||||
text: string,
|
||||
options?: { images?: ImageContent[] },
|
||||
): Promise<AssistantMessage> {
|
||||
this.applyTurnState(turnState);
|
||||
const beforeLength = this.agent.state.messages.length;
|
||||
let activeTurnState = turnState;
|
||||
let messages: AgentMessage[] = [createUserMessage(text, options?.images)];
|
||||
if (this.nextTurnQueue.length > 0) {
|
||||
messages = [...this.nextTurnQueue, messages[0]!];
|
||||
@@ -442,41 +491,70 @@ export class AgentHarness<
|
||||
systemPrompt: turnState.systemPrompt,
|
||||
resources: turnState.resources,
|
||||
});
|
||||
if (beforeResult?.messages) messages = [...beforeResult.messages, ...messages];
|
||||
if (beforeResult?.systemPrompt) this.agent.state.systemPrompt = beforeResult.systemPrompt;
|
||||
if (beforeResult?.messages) messages = [...messages, ...beforeResult.messages];
|
||||
|
||||
const abortController = new AbortController();
|
||||
const getTurnState = () => activeTurnState;
|
||||
const setTurnState = (nextTurnState: AgentHarnessTurnState<TSkill, TPromptTemplate, TTool>) => {
|
||||
activeTurnState = nextTurnState;
|
||||
};
|
||||
this.runAbortController = abortController;
|
||||
const runResultPromise = (async () => {
|
||||
try {
|
||||
return await runAgentLoop(
|
||||
messages,
|
||||
this.createContext(turnState, beforeResult?.systemPrompt),
|
||||
this.createLoopConfig(getTurnState, setTurnState),
|
||||
(event) => this.handleAgentEvent(event, abortController.signal),
|
||||
abortController.signal,
|
||||
this.createStreamFn(getTurnState),
|
||||
);
|
||||
} catch (error) {
|
||||
return await this.emitRunFailure(
|
||||
activeTurnState.model,
|
||||
error,
|
||||
abortController.signal.aborted,
|
||||
abortController.signal,
|
||||
);
|
||||
}
|
||||
})();
|
||||
try {
|
||||
await this.agent.prompt(messages);
|
||||
const newMessages = await runResultPromise;
|
||||
for (let i = newMessages.length - 1; i >= 0; i--) {
|
||||
const message = newMessages[i]!;
|
||||
if (message.role === "assistant") {
|
||||
return message;
|
||||
}
|
||||
}
|
||||
throw new Error("AgentHarness prompt completed without an assistant message");
|
||||
} finally {
|
||||
await this.flushPendingSessionWrites();
|
||||
}
|
||||
let response: AssistantMessage | undefined;
|
||||
const newMessages = this.agent.state.messages.slice(beforeLength);
|
||||
for (let i = newMessages.length - 1; i >= 0; i--) {
|
||||
const message = newMessages[i]!;
|
||||
if (message.role === "assistant") {
|
||||
response = message;
|
||||
break;
|
||||
try {
|
||||
await this.flushPendingSessionWrites();
|
||||
} finally {
|
||||
this.runAbortController = undefined;
|
||||
}
|
||||
}
|
||||
if (!response) throw new Error("AgentHarness prompt completed without an assistant message");
|
||||
return response;
|
||||
}
|
||||
|
||||
async prompt(text: string, options?: { images?: ImageContent[] }): Promise<AssistantMessage> {
|
||||
if (this.phase !== "idle") throw new Error("AgentHarness is busy");
|
||||
this.phase = "turn";
|
||||
const finishRunPromise = this.startRunPromise();
|
||||
try {
|
||||
const turnState = await this.createTurnState();
|
||||
return await this.executeTurn(turnState, text, options);
|
||||
} catch (error) {
|
||||
this.phase = "idle";
|
||||
throw error;
|
||||
} finally {
|
||||
finishRunPromise();
|
||||
}
|
||||
}
|
||||
|
||||
async skill(name: string, additionalInstructions?: string): Promise<AssistantMessage> {
|
||||
if (this.phase !== "idle") throw new Error("AgentHarness is busy");
|
||||
this.phase = "turn";
|
||||
const finishRunPromise = this.startRunPromise();
|
||||
try {
|
||||
const turnState = await this.createTurnState();
|
||||
const skill = (turnState.resources.skills ?? []).find((candidate) => candidate.name === name);
|
||||
@@ -485,12 +563,15 @@ export class AgentHarness<
|
||||
} catch (error) {
|
||||
this.phase = "idle";
|
||||
throw error;
|
||||
} finally {
|
||||
finishRunPromise();
|
||||
}
|
||||
}
|
||||
|
||||
async promptFromTemplate(name: string, args: string[] = []): Promise<AssistantMessage> {
|
||||
if (this.phase !== "idle") throw new Error("AgentHarness is busy");
|
||||
this.phase = "turn";
|
||||
const finishRunPromise = this.startRunPromise();
|
||||
try {
|
||||
const turnState = await this.createTurnState();
|
||||
const template = (turnState.resources.promptTemplates ?? []).find((candidate) => candidate.name === name);
|
||||
@@ -499,22 +580,20 @@ export class AgentHarness<
|
||||
} catch (error) {
|
||||
this.phase = "idle";
|
||||
throw error;
|
||||
} finally {
|
||||
finishRunPromise();
|
||||
}
|
||||
}
|
||||
|
||||
steer(text: string, options?: { images?: ImageContent[] }): void {
|
||||
if (this.phase === "idle") throw new Error("Cannot steer while idle");
|
||||
const message = createUserMessage(text, options?.images);
|
||||
this.steerQueue.push(message);
|
||||
this.agent.steer(message);
|
||||
this.steerQueue.push(createUserMessage(text, options?.images));
|
||||
void this.emitQueueUpdate();
|
||||
}
|
||||
|
||||
followUp(text: string, options?: { images?: ImageContent[] }): void {
|
||||
if (this.phase === "idle") throw new Error("Cannot follow up while idle");
|
||||
const message = createUserMessage(text, options?.images);
|
||||
this.followUpQueue.push(message);
|
||||
this.agent.followUp(message);
|
||||
this.followUpQueue.push(createUserMessage(text, options?.images));
|
||||
void this.emitQueueUpdate();
|
||||
}
|
||||
|
||||
@@ -690,11 +769,18 @@ export class AgentHarness<
|
||||
return { cancelled: false, editorText, summaryEntry };
|
||||
}
|
||||
|
||||
getModel(): Model<any> {
|
||||
return this.model;
|
||||
}
|
||||
|
||||
getThinkingLevel(): ThinkingLevel {
|
||||
return this.thinkingLevel;
|
||||
}
|
||||
|
||||
async setModel(model: Model<any>): Promise<void> {
|
||||
const previousModel = this.model;
|
||||
this.model = model;
|
||||
if (this.phase === "idle") {
|
||||
this.agent.state.model = model;
|
||||
await this.session.appendModelChange(model.provider, model.id);
|
||||
} else {
|
||||
this.pendingSessionWrites.push({ type: "model_change", provider: model.provider, modelId: model.id });
|
||||
@@ -706,7 +792,6 @@ export class AgentHarness<
|
||||
const previousLevel = this.thinkingLevel;
|
||||
this.thinkingLevel = level;
|
||||
if (this.phase === "idle") {
|
||||
this.agent.state.thinkingLevel = level;
|
||||
await this.session.appendThinkingLevelChange(level);
|
||||
} else {
|
||||
this.pendingSessionWrites.push({ type: "thinking_level_change", thinkingLevel: level });
|
||||
@@ -717,25 +802,22 @@ export class AgentHarness<
|
||||
async setActiveTools(toolNames: string[]): Promise<void> {
|
||||
this.validateToolNames(toolNames);
|
||||
this.activeToolNames = [...toolNames];
|
||||
if (this.phase === "idle") {
|
||||
this.agent.state.tools = this.activeToolNames.map((name) => this.tools.get(name)!);
|
||||
}
|
||||
}
|
||||
|
||||
get steeringMode(): QueueMode {
|
||||
return this.agent.steeringMode;
|
||||
getSteeringMode(): QueueMode {
|
||||
return this.steeringQueueMode;
|
||||
}
|
||||
|
||||
set steeringMode(mode: QueueMode) {
|
||||
this.agent.steeringMode = mode;
|
||||
setSteeringMode(mode: QueueMode): void {
|
||||
this.steeringQueueMode = mode;
|
||||
}
|
||||
|
||||
get followUpMode(): QueueMode {
|
||||
return this.agent.followUpMode;
|
||||
getFollowUpMode(): QueueMode {
|
||||
return this.followUpQueueMode;
|
||||
}
|
||||
|
||||
set followUpMode(mode: QueueMode) {
|
||||
this.agent.followUpMode = mode;
|
||||
setFollowUpMode(mode: QueueMode): void {
|
||||
this.followUpQueueMode = mode;
|
||||
}
|
||||
|
||||
getResources(): AgentHarnessResources<TSkill, TPromptTemplate> {
|
||||
@@ -770,9 +852,6 @@ export class AgentHarness<
|
||||
} else {
|
||||
this.validateToolNames(this.activeToolNames);
|
||||
}
|
||||
if (this.phase === "idle") {
|
||||
this.agent.state.tools = this.activeToolNames.map((name) => this.tools.get(name)!);
|
||||
}
|
||||
}
|
||||
|
||||
async abort(): Promise<AbortResult> {
|
||||
@@ -780,23 +859,27 @@ export class AgentHarness<
|
||||
const clearedFollowUp = [...this.followUpQueue];
|
||||
this.steerQueue = [];
|
||||
this.followUpQueue = [];
|
||||
this.agent.clearAllQueues();
|
||||
await this.emitQueueUpdate();
|
||||
this.agent.abort();
|
||||
await this.agent.waitForIdle();
|
||||
this.runAbortController?.abort();
|
||||
await this.waitForIdle();
|
||||
await this.emitOwn({ type: "abort", clearedSteer, clearedFollowUp });
|
||||
return { clearedSteer, clearedFollowUp };
|
||||
}
|
||||
|
||||
async waitForIdle(): Promise<void> {
|
||||
await this.agent.waitForIdle();
|
||||
await this.runPromise;
|
||||
}
|
||||
|
||||
subscribe(
|
||||
listener: (event: AgentHarnessEvent<TSkill, TPromptTemplate>, signal?: AbortSignal) => Promise<void> | void,
|
||||
): () => void {
|
||||
this.listeners.add(listener);
|
||||
return () => this.listeners.delete(listener);
|
||||
let handlers = this.handlers.get(SUBSCRIBER_EVENT_TYPE);
|
||||
if (!handlers) {
|
||||
handlers = new Set();
|
||||
this.handlers.set(SUBSCRIBER_EVENT_TYPE, handlers);
|
||||
}
|
||||
handlers.add(listener as AgentHarnessHandler);
|
||||
return () => handlers!.delete(listener as AgentHarnessHandler);
|
||||
}
|
||||
|
||||
on<TType extends keyof AgentHarnessEventResultMap>(
|
||||
@@ -805,12 +888,12 @@ export class AgentHarness<
|
||||
event: Extract<AgentHarnessOwnEvent, { type: TType }>,
|
||||
) => Promise<AgentHarnessEventResultMap[TType]> | AgentHarnessEventResultMap[TType],
|
||||
): () => void {
|
||||
let handlers = this.hooks.get(type);
|
||||
let handlers = this.handlers.get(type);
|
||||
if (!handlers) {
|
||||
handlers = new Set();
|
||||
this.hooks.set(type, handlers);
|
||||
this.handlers.set(type, handlers);
|
||||
}
|
||||
handlers.add(handler as any);
|
||||
return () => handlers!.delete(handler as any);
|
||||
handlers.add(handler as AgentHarnessHandler);
|
||||
return () => handlers!.delete(handler as AgentHarnessHandler);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import type { ImageContent, Model, SimpleStreamOptions, TextContent, Transport } from "@earendil-works/pi-ai";
|
||||
import type { QueueMode } from "../agent.js";
|
||||
import type { AgentEvent, AgentMessage, AgentTool, ThinkingLevel } from "../index.js";
|
||||
import type { AgentEvent, AgentMessage, AgentTool, QueueMode, ThinkingLevel } from "../index.js";
|
||||
import type { Session } from "./session/session.js";
|
||||
|
||||
/** Result of a fallible operation. Expected failures are returned as `ok: false` instead of thrown. */
|
||||
|
||||
@@ -44,6 +44,64 @@ export interface TruncationOptions {
|
||||
maxBytes?: number;
|
||||
}
|
||||
|
||||
interface RuntimeBuffer {
|
||||
byteLength(content: string, encoding: "utf8"): number;
|
||||
}
|
||||
|
||||
const runtimeBuffer = (globalThis as { Buffer?: RuntimeBuffer }).Buffer;
|
||||
const nonAsciiPattern = /[^\x00-\x7f]/;
|
||||
|
||||
function utf8ByteLength(content: string): number {
|
||||
if (runtimeBuffer) return runtimeBuffer.byteLength(content, "utf8");
|
||||
|
||||
const firstNonAscii = content.search(nonAsciiPattern);
|
||||
if (firstNonAscii === -1) return content.length;
|
||||
|
||||
let bytes = firstNonAscii;
|
||||
for (let i = firstNonAscii; i < content.length; i++) {
|
||||
const code = content.charCodeAt(i);
|
||||
if (code <= 0x7f) {
|
||||
bytes += 1;
|
||||
} else if (code <= 0x7ff) {
|
||||
bytes += 2;
|
||||
} else if (code >= 0xd800 && code <= 0xdbff && i + 1 < content.length) {
|
||||
const next = content.charCodeAt(i + 1);
|
||||
if (next >= 0xdc00 && next <= 0xdfff) {
|
||||
bytes += 4;
|
||||
i++;
|
||||
} else {
|
||||
bytes += 3;
|
||||
}
|
||||
} else {
|
||||
bytes += 3;
|
||||
}
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
|
||||
function replaceUnpairedSurrogates(content: string): string {
|
||||
let output = "";
|
||||
for (let i = 0; i < content.length; i++) {
|
||||
const code = content.charCodeAt(i);
|
||||
if (code >= 0xd800 && code <= 0xdbff) {
|
||||
if (i + 1 < content.length) {
|
||||
const next = content.charCodeAt(i + 1);
|
||||
if (next >= 0xdc00 && next <= 0xdfff) {
|
||||
output += content[i] + content[i + 1];
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
output += "<22>";
|
||||
} else if (code >= 0xdc00 && code <= 0xdfff) {
|
||||
output += "<22>";
|
||||
} else {
|
||||
output += content[i];
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
/**
|
||||
* Format bytes as human-readable size.
|
||||
*/
|
||||
@@ -68,7 +126,7 @@ export function truncateHead(content: string, options: TruncationOptions = {}):
|
||||
const maxLines = options.maxLines ?? DEFAULT_MAX_LINES;
|
||||
const maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES;
|
||||
|
||||
const totalBytes = Buffer.byteLength(content, "utf-8");
|
||||
const totalBytes = utf8ByteLength(content);
|
||||
const lines = content.split("\n");
|
||||
const totalLines = lines.length;
|
||||
|
||||
@@ -90,7 +148,7 @@ export function truncateHead(content: string, options: TruncationOptions = {}):
|
||||
}
|
||||
|
||||
// Check if first line alone exceeds byte limit
|
||||
const firstLineBytes = Buffer.byteLength(lines[0], "utf-8");
|
||||
const firstLineBytes = utf8ByteLength(lines[0]);
|
||||
if (firstLineBytes > maxBytes) {
|
||||
return {
|
||||
content: "",
|
||||
@@ -114,7 +172,7 @@ export function truncateHead(content: string, options: TruncationOptions = {}):
|
||||
|
||||
for (let i = 0; i < lines.length && i < maxLines; i++) {
|
||||
const line = lines[i];
|
||||
const lineBytes = Buffer.byteLength(line, "utf-8") + (i > 0 ? 1 : 0); // +1 for newline
|
||||
const lineBytes = utf8ByteLength(line) + (i > 0 ? 1 : 0); // +1 for newline
|
||||
|
||||
if (outputBytesCount + lineBytes > maxBytes) {
|
||||
truncatedBy = "bytes";
|
||||
@@ -131,7 +189,7 @@ export function truncateHead(content: string, options: TruncationOptions = {}):
|
||||
}
|
||||
|
||||
const outputContent = outputLinesArr.join("\n");
|
||||
const finalOutputBytes = Buffer.byteLength(outputContent, "utf-8");
|
||||
const finalOutputBytes = utf8ByteLength(outputContent);
|
||||
|
||||
return {
|
||||
content: outputContent,
|
||||
@@ -158,7 +216,7 @@ export function truncateTail(content: string, options: TruncationOptions = {}):
|
||||
const maxLines = options.maxLines ?? DEFAULT_MAX_LINES;
|
||||
const maxBytes = options.maxBytes ?? DEFAULT_MAX_BYTES;
|
||||
|
||||
const totalBytes = Buffer.byteLength(content, "utf-8");
|
||||
const totalBytes = utf8ByteLength(content);
|
||||
const lines = content.split("\n");
|
||||
const totalLines = lines.length;
|
||||
|
||||
@@ -187,7 +245,7 @@ export function truncateTail(content: string, options: TruncationOptions = {}):
|
||||
|
||||
for (let i = lines.length - 1; i >= 0 && outputLinesArr.length < maxLines; i--) {
|
||||
const line = lines[i];
|
||||
const lineBytes = Buffer.byteLength(line, "utf-8") + (outputLinesArr.length > 0 ? 1 : 0); // +1 for newline
|
||||
const lineBytes = utf8ByteLength(line) + (outputLinesArr.length > 0 ? 1 : 0); // +1 for newline
|
||||
|
||||
if (outputBytesCount + lineBytes > maxBytes) {
|
||||
truncatedBy = "bytes";
|
||||
@@ -196,7 +254,7 @@ export function truncateTail(content: string, options: TruncationOptions = {}):
|
||||
if (outputLinesArr.length === 0) {
|
||||
const truncatedLine = truncateStringToBytesFromEnd(line, maxBytes);
|
||||
outputLinesArr.unshift(truncatedLine);
|
||||
outputBytesCount = Buffer.byteLength(truncatedLine, "utf-8");
|
||||
outputBytesCount = utf8ByteLength(truncatedLine);
|
||||
lastLinePartial = true;
|
||||
}
|
||||
break;
|
||||
@@ -212,7 +270,7 @@ export function truncateTail(content: string, options: TruncationOptions = {}):
|
||||
}
|
||||
|
||||
const outputContent = outputLinesArr.join("\n");
|
||||
const finalOutputBytes = Buffer.byteLength(outputContent, "utf-8");
|
||||
const finalOutputBytes = utf8ByteLength(outputContent);
|
||||
|
||||
return {
|
||||
content: outputContent,
|
||||
@@ -234,20 +292,40 @@ export function truncateTail(content: string, options: TruncationOptions = {}):
|
||||
* Handles multi-byte UTF-8 characters correctly.
|
||||
*/
|
||||
function truncateStringToBytesFromEnd(str: string, maxBytes: number): string {
|
||||
const buf = Buffer.from(str, "utf-8");
|
||||
if (buf.length <= maxBytes) {
|
||||
return str;
|
||||
if (maxBytes <= 0) return "";
|
||||
|
||||
let outputBytes = 0;
|
||||
let start = str.length;
|
||||
let needsReplacement = false;
|
||||
for (let i = str.length; i > 0; ) {
|
||||
let characterStart = i - 1;
|
||||
const code = str.charCodeAt(characterStart);
|
||||
let characterBytes: number;
|
||||
let unpairedSurrogate = false;
|
||||
if (code >= 0xdc00 && code <= 0xdfff && characterStart > 0) {
|
||||
const previous = str.charCodeAt(characterStart - 1);
|
||||
if (previous >= 0xd800 && previous <= 0xdbff) {
|
||||
characterStart--;
|
||||
characterBytes = 4;
|
||||
} else {
|
||||
characterBytes = 3;
|
||||
unpairedSurrogate = true;
|
||||
}
|
||||
} else if (code >= 0xd800 && code <= 0xdfff) {
|
||||
characterBytes = 3;
|
||||
unpairedSurrogate = true;
|
||||
} else {
|
||||
characterBytes = code <= 0x7f ? 1 : code <= 0x7ff ? 2 : 3;
|
||||
}
|
||||
if (outputBytes + characterBytes > maxBytes) break;
|
||||
outputBytes += characterBytes;
|
||||
start = characterStart;
|
||||
needsReplacement ||= unpairedSurrogate;
|
||||
i = characterStart;
|
||||
}
|
||||
|
||||
// Start from the end, skip maxBytes back
|
||||
let start = buf.length - maxBytes;
|
||||
|
||||
// Find a valid UTF-8 boundary (start of a character)
|
||||
while (start < buf.length && (buf[start] & 0xc0) === 0x80) {
|
||||
start++;
|
||||
}
|
||||
|
||||
return buf.slice(start).toString("utf-8");
|
||||
const output = str.slice(start);
|
||||
return needsReplacement ? replaceUnpairedSurrogates(output) : output;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -35,6 +35,14 @@ export type StreamFn = (
|
||||
*/
|
||||
export type ToolExecutionMode = "sequential" | "parallel";
|
||||
|
||||
/**
|
||||
* Controls how many queued user messages are injected when the agent loop reaches a queue drain point.
|
||||
*
|
||||
* - "all": drain and inject every queued message at that point.
|
||||
* - "one-at-a-time": drain and inject only the oldest queued message, leaving the rest queued for later drain points.
|
||||
*/
|
||||
export type QueueMode = "all" | "one-at-a-time";
|
||||
|
||||
/** A single tool call content block emitted by an assistant message. */
|
||||
export type AgentToolCall = Extract<AssistantMessage["content"][number], { type: "toolCall" }>;
|
||||
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import { getModel } from "@earendil-works/pi-ai";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { fauxAssistantMessage, fauxToolCall, getModel, registerFauxProvider } from "@earendil-works/pi-ai";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { AgentHarness } from "../../src/harness/agent-harness.js";
|
||||
import { NodeExecutionEnv } from "../../src/harness/env/nodejs.js";
|
||||
import { Session } from "../../src/harness/session/session.js";
|
||||
import { InMemorySessionStorage } from "../../src/harness/session/storage/memory.js";
|
||||
import type { PromptTemplate, Skill } from "../../src/harness/types.js";
|
||||
import type { AgentTool } from "../../src/types.js";
|
||||
import type { AgentMessage, AgentTool } from "../../src/types.js";
|
||||
import { calculateTool } from "../utils/calculate.js";
|
||||
import { getCurrentTimeTool } from "../utils/get-current-time.js";
|
||||
|
||||
interface AppSkill extends Skill {
|
||||
source: "project" | "user";
|
||||
@@ -19,6 +21,39 @@ interface AppTool extends AgentTool {
|
||||
source: "builtin" | "extension";
|
||||
}
|
||||
|
||||
const registrations: Array<{ unregister(): void }> = [];
|
||||
|
||||
function textFromUserMessages(messages: Array<{ role: string; content: unknown }>): string[] {
|
||||
return messages.flatMap((message) => {
|
||||
if (message.role !== "user") return [];
|
||||
if (typeof message.content === "string") return [message.content];
|
||||
if (!Array.isArray(message.content)) return [];
|
||||
return message.content.flatMap((part) => {
|
||||
if (!part || typeof part !== "object" || !("type" in part) || part.type !== "text") return [];
|
||||
return "text" in part && typeof part.text === "string" ? [part.text] : [];
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function deferred(): { promise: Promise<void>; resolve: () => void } {
|
||||
let resolve = () => {};
|
||||
const promise = new Promise<void>((resolvePromise) => {
|
||||
resolve = resolvePromise;
|
||||
});
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
function getReasoning(options: unknown): unknown {
|
||||
if (!options || typeof options !== "object" || !("reasoning" in options)) return undefined;
|
||||
return options.reasoning;
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const registration of registrations.splice(0)) {
|
||||
registration.unregister();
|
||||
}
|
||||
});
|
||||
|
||||
describe("AgentHarness", () => {
|
||||
it("constructs directly and exposes queue modes", () => {
|
||||
const session = new Session(new InMemorySessionStorage());
|
||||
@@ -28,18 +63,399 @@ describe("AgentHarness", () => {
|
||||
env,
|
||||
session,
|
||||
model: initialModel,
|
||||
thinkingLevel: "high",
|
||||
systemPrompt: "You are helpful.",
|
||||
steeringMode: "all",
|
||||
followUpMode: "all",
|
||||
});
|
||||
expect(harness.env).toBe(env);
|
||||
expect(harness.agent.state.model).toBe(initialModel);
|
||||
expect(harness.steeringMode).toBe("all");
|
||||
expect(harness.followUpMode).toBe("all");
|
||||
harness.steeringMode = "one-at-a-time";
|
||||
harness.followUpMode = "one-at-a-time";
|
||||
expect(harness.agent.steeringMode).toBe("one-at-a-time");
|
||||
expect(harness.agent.followUpMode).toBe("one-at-a-time");
|
||||
expect(harness.getModel()).toBe(initialModel);
|
||||
expect(harness.getThinkingLevel()).toBe("high");
|
||||
expect(harness.getSteeringMode()).toBe("all");
|
||||
expect(harness.getFollowUpMode()).toBe("all");
|
||||
harness.setSteeringMode("one-at-a-time");
|
||||
harness.setFollowUpMode("one-at-a-time");
|
||||
expect(harness.getSteeringMode()).toBe("one-at-a-time");
|
||||
expect(harness.getFollowUpMode()).toBe("one-at-a-time");
|
||||
});
|
||||
|
||||
it("drains one queued steering message at a time and emits queue updates", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
const userCounts: number[] = [];
|
||||
registration.setResponses([
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("first");
|
||||
},
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("second");
|
||||
},
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("third");
|
||||
},
|
||||
]);
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session: new Session(new InMemorySessionStorage()),
|
||||
model: registration.getModel(),
|
||||
steeringMode: "one-at-a-time",
|
||||
});
|
||||
const steerQueueLengths: number[] = [];
|
||||
let queued = false;
|
||||
harness.subscribe((event) => {
|
||||
if (event.type === "queue_update") {
|
||||
steerQueueLengths.push(event.steer.length);
|
||||
}
|
||||
if (event.type === "message_start" && event.message.role === "assistant" && !queued) {
|
||||
queued = true;
|
||||
harness.steer("one");
|
||||
harness.steer("two");
|
||||
}
|
||||
});
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
expect(userCounts).toEqual([1, 2, 3]);
|
||||
expect(steerQueueLengths).toEqual([1, 2, 1, 0]);
|
||||
});
|
||||
|
||||
it("appends before_agent_start messages and persists them", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
let requestText: string[] = [];
|
||||
registration.setResponses([
|
||||
(context) => {
|
||||
requestText = textFromUserMessages(context.messages);
|
||||
return fauxAssistantMessage("ok");
|
||||
},
|
||||
]);
|
||||
const session = new Session(new InMemorySessionStorage());
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session,
|
||||
model: registration.getModel(),
|
||||
});
|
||||
harness.on("before_agent_start", () => ({
|
||||
messages: [{ role: "user", content: [{ type: "text", text: "hook" }], timestamp: Date.now() }],
|
||||
}));
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
const persistedText = (await session.getEntries()).flatMap((entry) => {
|
||||
if (entry.type !== "message" || entry.message.role !== "user") return [];
|
||||
const content = entry.message.content;
|
||||
if (typeof content === "string") return [content];
|
||||
return content.flatMap((part) => (part.type === "text" ? [part.text] : []));
|
||||
});
|
||||
expect(requestText).toEqual(["hello", "hook"]);
|
||||
expect(persistedText).toEqual(["hello", "hook"]);
|
||||
});
|
||||
|
||||
it("abort clears steer and follow-up queues but preserves next-turn messages", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
let releaseFirstResponse: (() => void) | undefined;
|
||||
let abortedSignal: AbortSignal | undefined;
|
||||
const firstResponseReleased = new Promise<void>((resolve) => {
|
||||
releaseFirstResponse = resolve;
|
||||
});
|
||||
const secondRequestText: string[] = [];
|
||||
registration.setResponses([
|
||||
async (_context, options) => {
|
||||
abortedSignal = options?.signal;
|
||||
await firstResponseReleased;
|
||||
return fauxAssistantMessage("aborted-ish");
|
||||
},
|
||||
(context) => {
|
||||
secondRequestText.push(...textFromUserMessages(context.messages));
|
||||
return fauxAssistantMessage("second");
|
||||
},
|
||||
]);
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session: new Session(new InMemorySessionStorage()),
|
||||
model: registration.getModel(),
|
||||
});
|
||||
const queueUpdates: Array<{ steer: number; followUp: number; nextTurn: number }> = [];
|
||||
harness.subscribe((event) => {
|
||||
if (event.type === "queue_update") {
|
||||
queueUpdates.push({
|
||||
steer: event.steer.length,
|
||||
followUp: event.followUp.length,
|
||||
nextTurn: event.nextTurn.length,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
const firstPrompt = harness.prompt("first");
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
harness.steer("steer");
|
||||
harness.followUp("follow");
|
||||
harness.nextTurn("next");
|
||||
const abortResultPromise = harness.abort();
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(abortedSignal?.aborted).toBe(true);
|
||||
releaseFirstResponse?.();
|
||||
const abortResult = await abortResultPromise;
|
||||
await firstPrompt;
|
||||
await harness.prompt("second");
|
||||
|
||||
expect(abortResult.clearedSteer).toHaveLength(1);
|
||||
expect(abortResult.clearedFollowUp).toHaveLength(1);
|
||||
expect(queueUpdates).toContainEqual({ steer: 0, followUp: 0, nextTurn: 1 });
|
||||
expect(secondRequestText).toEqual(["first", "next", "second"]);
|
||||
});
|
||||
|
||||
it("drains follow-up messages one at a time after the agent would otherwise stop", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
const userCounts: number[] = [];
|
||||
registration.setResponses([
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("first");
|
||||
},
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("second");
|
||||
},
|
||||
(context) => {
|
||||
userCounts.push(context.messages.filter((message) => message.role === "user").length);
|
||||
return fauxAssistantMessage("third");
|
||||
},
|
||||
]);
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session: new Session(new InMemorySessionStorage()),
|
||||
model: registration.getModel(),
|
||||
followUpMode: "one-at-a-time",
|
||||
});
|
||||
const followUpQueueLengths: number[] = [];
|
||||
let queued = false;
|
||||
harness.subscribe((event) => {
|
||||
if (event.type === "queue_update") {
|
||||
followUpQueueLengths.push(event.followUp.length);
|
||||
}
|
||||
if (event.type === "message_start" && event.message.role === "assistant" && !queued) {
|
||||
queued = true;
|
||||
harness.followUp("one");
|
||||
harness.followUp("two");
|
||||
}
|
||||
});
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
expect(userCounts).toEqual([1, 2, 3]);
|
||||
expect(followUpQueueLengths).toEqual([1, 2, 1, 0]);
|
||||
});
|
||||
|
||||
it("settles thrown hook failures with persisted assistant error messages", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
registration.setResponses([() => fauxAssistantMessage("should not be used")]);
|
||||
const session = new Session(new InMemorySessionStorage());
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session,
|
||||
model: registration.getModel(),
|
||||
});
|
||||
const events: string[] = [];
|
||||
harness.subscribe((event) => {
|
||||
events.push(event.type);
|
||||
});
|
||||
harness.on("context", () => {
|
||||
throw new Error("context exploded");
|
||||
});
|
||||
|
||||
const response = await harness.prompt("hello");
|
||||
await expect(harness.prompt("after failure")).resolves.toMatchObject({ role: "assistant" });
|
||||
|
||||
const entries = await session.getEntries();
|
||||
const messages = entries.flatMap((entry) => (entry.type === "message" ? [entry.message] : []));
|
||||
expect(response.stopReason).toBe("error");
|
||||
expect(response.errorMessage).toBe("context exploded");
|
||||
expect(messages[0]?.role).toBe("user");
|
||||
expect(messages[1]).toMatchObject({ role: "assistant", stopReason: "error", errorMessage: "context exploded" });
|
||||
expect(events).toContain("agent_end");
|
||||
expect(events).toContain("settled");
|
||||
});
|
||||
|
||||
it("refreshes model, thinking level, resources, system prompt, and active tools at save points", async () => {
|
||||
const registration = registerFauxProvider({
|
||||
models: [
|
||||
{ id: "first", reasoning: true },
|
||||
{ id: "second", reasoning: true },
|
||||
],
|
||||
});
|
||||
registrations.push(registration);
|
||||
const secondModel = registration.getModel("second");
|
||||
if (!secondModel) throw new Error("missing second faux model");
|
||||
const captured: Array<{ modelId: string; reasoning: unknown; systemPrompt: string; tools: string[] }> = [];
|
||||
registration.setResponses([
|
||||
(context, options, _state, model) => {
|
||||
captured.push({
|
||||
modelId: model.id,
|
||||
reasoning: getReasoning(options),
|
||||
systemPrompt: context.systemPrompt ?? "",
|
||||
tools: context.tools?.map((tool) => tool.name) ?? [],
|
||||
});
|
||||
return fauxAssistantMessage(fauxToolCall("calculate", { expression: "1 + 1" }, { id: "call-1" }), {
|
||||
stopReason: "toolUse",
|
||||
});
|
||||
},
|
||||
(context, options, _state, model) => {
|
||||
captured.push({
|
||||
modelId: model.id,
|
||||
reasoning: getReasoning(options),
|
||||
systemPrompt: context.systemPrompt ?? "",
|
||||
tools: context.tools?.map((tool) => tool.name) ?? [],
|
||||
});
|
||||
return fauxAssistantMessage("done");
|
||||
},
|
||||
]);
|
||||
const harness = new AgentHarness<Skill, PromptTemplate, AgentTool>({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session: new Session(new InMemorySessionStorage()),
|
||||
model: registration.getModel(),
|
||||
thinkingLevel: "off",
|
||||
resources: {
|
||||
skills: [{ name: "prompt", description: "prompt", content: "first prompt", filePath: "/skills/prompt" }],
|
||||
},
|
||||
systemPrompt: ({ resources }) => resources.skills?.[0]?.content ?? "missing prompt",
|
||||
tools: [calculateTool],
|
||||
});
|
||||
harness.subscribe((event) => {
|
||||
if (event.type === "tool_execution_start") {
|
||||
void harness.setModel(secondModel);
|
||||
void harness.setThinkingLevel("high");
|
||||
void harness.setResources({
|
||||
skills: [
|
||||
{ name: "prompt", description: "prompt", content: "second prompt", filePath: "/skills/prompt" },
|
||||
],
|
||||
});
|
||||
void harness.setTools([calculateTool, getCurrentTimeTool], [getCurrentTimeTool.name]);
|
||||
}
|
||||
});
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
expect(captured).toEqual([
|
||||
{ modelId: "first", reasoning: undefined, systemPrompt: "first prompt", tools: ["calculate"] },
|
||||
{ modelId: "second", reasoning: "high", systemPrompt: "second prompt", tools: ["get_current_time"] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("orders pending listener session writes after agent-emitted messages", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
registration.setResponses([() => fauxAssistantMessage("ok")]);
|
||||
const session = new Session(new InMemorySessionStorage());
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session,
|
||||
model: registration.getModel(),
|
||||
});
|
||||
let wrotePendingMessage = false;
|
||||
harness.subscribe(async (event) => {
|
||||
if (event.type === "message_end" && event.message.role === "assistant" && !wrotePendingMessage) {
|
||||
wrotePendingMessage = true;
|
||||
await harness.appendMessage({
|
||||
role: "custom",
|
||||
customType: "listener",
|
||||
content: "listener write",
|
||||
display: true,
|
||||
timestamp: Date.now(),
|
||||
} as AgentMessage);
|
||||
}
|
||||
});
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
const entries = await session.getEntries();
|
||||
const roles = entries.flatMap((entry) => (entry.type === "message" ? [entry.message.role] : []));
|
||||
expect(roles).toEqual(["user", "assistant", "custom"]);
|
||||
});
|
||||
|
||||
it("waitForIdle waits for external run settlement and awaited listeners", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
registration.setResponses([() => fauxAssistantMessage("ok")]);
|
||||
const barrier = deferred();
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session: new Session(new InMemorySessionStorage()),
|
||||
model: registration.getModel(),
|
||||
});
|
||||
let listenerFinished = false;
|
||||
harness.subscribe(async (event) => {
|
||||
if (event.type === "agent_end") {
|
||||
await barrier.promise;
|
||||
listenerFinished = true;
|
||||
}
|
||||
});
|
||||
|
||||
const promptPromise = harness.prompt("hello");
|
||||
let idleResolved = false;
|
||||
const idlePromise = harness.waitForIdle().then(() => {
|
||||
idleResolved = true;
|
||||
});
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
expect(idleResolved).toBe(false);
|
||||
expect(listenerFinished).toBe(false);
|
||||
barrier.resolve();
|
||||
await Promise.all([promptPromise, idlePromise]);
|
||||
expect(idleResolved).toBe(true);
|
||||
expect(listenerFinished).toBe(true);
|
||||
});
|
||||
|
||||
it("runs tool_call and tool_result hooks through the direct loop", async () => {
|
||||
const registration = registerFauxProvider();
|
||||
registrations.push(registration);
|
||||
registration.setResponses([
|
||||
() =>
|
||||
fauxAssistantMessage(fauxToolCall("calculate", { expression: "2 + 2" }, { id: "call-1" }), {
|
||||
stopReason: "toolUse",
|
||||
}),
|
||||
]);
|
||||
const session = new Session(new InMemorySessionStorage());
|
||||
const harness = new AgentHarness({
|
||||
env: new NodeExecutionEnv({ cwd: process.cwd() }),
|
||||
session,
|
||||
model: registration.getModel(),
|
||||
tools: [calculateTool],
|
||||
});
|
||||
const seenToolCalls: Array<{ id: string; name: string; expression: unknown }> = [];
|
||||
harness.on("tool_call", (event) => {
|
||||
seenToolCalls.push({ id: event.toolCallId, name: event.toolName, expression: event.input.expression });
|
||||
return undefined;
|
||||
});
|
||||
harness.on("tool_result", (event) => {
|
||||
expect(event.toolCallId).toBe("call-1");
|
||||
expect(event.toolName).toBe("calculate");
|
||||
return {
|
||||
content: [{ type: "text", text: "patched result" }],
|
||||
details: { patched: true },
|
||||
terminate: true,
|
||||
};
|
||||
});
|
||||
|
||||
await harness.prompt("hello");
|
||||
|
||||
const toolResult = (await session.getEntries()).find(
|
||||
(entry) => entry.type === "message" && entry.message.role === "toolResult",
|
||||
);
|
||||
expect(seenToolCalls).toEqual([{ id: "call-1", name: "calculate", expression: "2 + 2" }]);
|
||||
expect(toolResult).toMatchObject({
|
||||
type: "message",
|
||||
message: {
|
||||
role: "toolResult",
|
||||
content: [{ type: "text", text: "patched result" }],
|
||||
details: { patched: true },
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves app resource types for getters and update events", async () => {
|
||||
|
||||
158
packages/agent/test/harness/truncate.test.ts
Normal file
158
packages/agent/test/harness/truncate.test.ts
Normal file
@@ -0,0 +1,158 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { truncateHead, truncateTail } from "../../src/harness/utils/truncate.js";
|
||||
|
||||
const encoder = new TextEncoder();
|
||||
|
||||
function byteLength(content: string): number {
|
||||
return encoder.encode(content).length;
|
||||
}
|
||||
|
||||
function bufferTail(content: string, maxBytes: number): string {
|
||||
const bytes = Buffer.from(content, "utf8");
|
||||
if (bytes.length <= maxBytes) return content;
|
||||
let start = bytes.length - maxBytes;
|
||||
while (start < bytes.length && (bytes[start] & 0xc0) === 0x80) start++;
|
||||
return bytes.subarray(start).toString("utf8");
|
||||
}
|
||||
|
||||
function assertMatchesBufferTail(input: string, maxByteValues?: readonly number[]): void {
|
||||
const totalBytes = Buffer.byteLength(input, "utf8");
|
||||
const values = maxByteValues ?? Array.from({ length: totalBytes + 5 }, (_, maxBytes) => maxBytes);
|
||||
for (const maxBytes of values) {
|
||||
const result = truncateTail(input, { maxBytes, maxLines: 10 });
|
||||
const expected = bufferTail(input, maxBytes);
|
||||
if (result.content !== expected) {
|
||||
throw new Error(
|
||||
`tail mismatch input=${JSON.stringify(input)} maxBytes=${maxBytes} expected=${JSON.stringify(expected)} actual=${JSON.stringify(result.content)}`,
|
||||
);
|
||||
}
|
||||
const outputBytes = Buffer.byteLength(result.content, "utf8");
|
||||
if (outputBytes > maxBytes) {
|
||||
throw new Error(
|
||||
`tail output exceeded byte limit input=${JSON.stringify(input)} maxBytes=${maxBytes} outputBytes=${outputBytes}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function sampledByteLimits(input: string): number[] {
|
||||
const totalBytes = Buffer.byteLength(input, "utf8");
|
||||
const candidates = [
|
||||
0,
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
8,
|
||||
Math.floor(totalBytes / 2) - 1,
|
||||
Math.floor(totalBytes / 2),
|
||||
Math.floor(totalBytes / 2) + 1,
|
||||
totalBytes - 8,
|
||||
totalBytes - 5,
|
||||
totalBytes - 4,
|
||||
totalBytes - 3,
|
||||
totalBytes - 2,
|
||||
totalBytes - 1,
|
||||
totalBytes,
|
||||
totalBytes + 1,
|
||||
totalBytes + 4,
|
||||
];
|
||||
return [...new Set(candidates.filter((value) => value >= 0))].sort((a, b) => a - b);
|
||||
}
|
||||
|
||||
describe("truncate utilities", () => {
|
||||
it("counts UTF-8 bytes without Node Buffer", () => {
|
||||
const content = "aé🙂\nb";
|
||||
const result = truncateHead(content, { maxBytes: 100, maxLines: 10 });
|
||||
|
||||
expect(result.truncated).toBe(false);
|
||||
expect(result.totalBytes).toBe(byteLength(content));
|
||||
expect(result.outputBytes).toBe(byteLength(content));
|
||||
expect(result.totalBytes).toBe(9);
|
||||
});
|
||||
|
||||
it("truncates head on UTF-8 byte limits without partial lines", () => {
|
||||
const content = "éé\nabc";
|
||||
const result = truncateHead(content, { maxBytes: 4, maxLines: 10 });
|
||||
|
||||
expect(result.content).toBe("éé");
|
||||
expect(result.truncated).toBe(true);
|
||||
expect(result.truncatedBy).toBe("bytes");
|
||||
expect(result.outputBytes).toBe(4);
|
||||
expect(result.firstLineExceedsLimit).toBe(false);
|
||||
});
|
||||
|
||||
it("reports head truncation when the first line exceeds the byte limit", () => {
|
||||
const result = truncateHead("éé\nabc", { maxBytes: 3, maxLines: 10 });
|
||||
|
||||
expect(result.content).toBe("");
|
||||
expect(result.truncated).toBe(true);
|
||||
expect(result.truncatedBy).toBe("bytes");
|
||||
expect(result.firstLineExceedsLimit).toBe(true);
|
||||
});
|
||||
|
||||
it("truncates tail on UTF-8 boundaries when only a partial last line fits", () => {
|
||||
const result = truncateTail("aé🙂b", { maxBytes: 5, maxLines: 10 });
|
||||
|
||||
expect(result.content).toBe("🙂b");
|
||||
expect(result.truncated).toBe(true);
|
||||
expect(result.truncatedBy).toBe("bytes");
|
||||
expect(result.lastLinePartial).toBe(true);
|
||||
expect(result.outputBytes).toBe(5);
|
||||
});
|
||||
|
||||
it("drops an oversized trailing character when it cannot fit in tail byte limit", () => {
|
||||
const result = truncateTail("abc🙂", { maxBytes: 3, maxLines: 10 });
|
||||
|
||||
expect(result.content).toBe("");
|
||||
expect(result.truncated).toBe(true);
|
||||
expect(result.truncatedBy).toBe("bytes");
|
||||
expect(result.lastLinePartial).toBe(true);
|
||||
expect(result.outputBytes).toBe(0);
|
||||
});
|
||||
|
||||
it("matches Buffer tail truncation semantics for surrogate edge cases", () => {
|
||||
const inputs = ["a\ud83d", "\ude42b", "a\ude42b", "\ud83d\ud83d\ude42", "\ud83d\ude42\ude42", "👩💻"];
|
||||
for (const input of inputs) assertMatchesBufferTail(input);
|
||||
});
|
||||
|
||||
it("matches Buffer tail truncation semantics across deterministic fuzz cases", () => {
|
||||
const alphabet = [
|
||||
"a",
|
||||
"\u007f",
|
||||
"\u0080",
|
||||
"é",
|
||||
"\u07ff",
|
||||
"\u0800",
|
||||
"中",
|
||||
"\ud7ff",
|
||||
"\ud800",
|
||||
"\ud83d",
|
||||
"\udc00",
|
||||
"\ude42",
|
||||
"🙂",
|
||||
"\ue000",
|
||||
"\uffff",
|
||||
];
|
||||
|
||||
function checkExhaustive(prefix: string, depth: number): void {
|
||||
assertMatchesBufferTail(prefix, sampledByteLimits(prefix));
|
||||
if (depth === 0) return;
|
||||
for (const character of alphabet) checkExhaustive(prefix + character, depth - 1);
|
||||
}
|
||||
checkExhaustive("", 3);
|
||||
|
||||
let seed = 0x12345678;
|
||||
function random(): number {
|
||||
seed = (seed * 1664525 + 1013904223) >>> 0;
|
||||
return seed / 0x100000000;
|
||||
}
|
||||
for (let i = 0; i < 1_000; i++) {
|
||||
let input = "";
|
||||
const length = Math.floor(random() * 80);
|
||||
for (let j = 0; j < length; j++) input += alphabet[Math.floor(random() * alphabet.length)];
|
||||
assertMatchesBufferTail(input, sampledByteLimits(input));
|
||||
}
|
||||
});
|
||||
});
|
||||
18
packages/agent/vitest.harness.config.ts
Normal file
18
packages/agent/vitest.harness.config.ts
Normal file
@@ -0,0 +1,18 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globals: true,
|
||||
environment: "node",
|
||||
testTimeout: 30000,
|
||||
include: ["test/harness/**/*.test.ts"],
|
||||
coverage: {
|
||||
provider: "v8",
|
||||
all: true,
|
||||
include: ["src/harness/**/*.ts", "src/agent.ts", "src/agent-loop.ts"],
|
||||
exclude: ["src/**/*.d.ts"],
|
||||
reporter: ["text", "html", "lcov"],
|
||||
reportsDirectory: "coverage/harness",
|
||||
},
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user