美化排版前端管理后台
This commit is contained in:
226
.claude/skills/trellis-channel/references/progress-debugging.md
Normal file
226
.claude/skills/trellis-channel/references/progress-debugging.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Progress And Debugging
|
||||
|
||||
Pretty output is for operators. Raw output is the audit log. Subcommands
|
||||
(`forum`, `thread`, `messages`, `context`) are the audit *interface* — reach
|
||||
for them before grepping `events.jsonl` by hand.
|
||||
|
||||
## Pretty vs `--raw`
|
||||
|
||||
`trellis channel messages <channel>` renders a compact, human-readable view:
|
||||
timestamps, identities, kind, and a short body. It is meant for operators
|
||||
scanning a channel, not for diagnostics.
|
||||
|
||||
Pretty output can and will truncate:
|
||||
|
||||
- long progress deltas (`text_delta`, partial tool args)
|
||||
- tool names and command lines
|
||||
- multi-line status fields and structured `detail` blobs
|
||||
- forum thread titles past the column budget
|
||||
|
||||
When something looks "off" — a worker appears stuck, a progress line ends
|
||||
mid-word, an action field shows `...` — switch to `--raw`. Raw mode emits
|
||||
one JSON event per line exactly as it lives in `events.jsonl`, so nothing
|
||||
is dropped.
|
||||
|
||||
```bash
|
||||
# Pretty (operator view)
|
||||
trellis channel messages <channel> --kind done --last 10
|
||||
trellis channel messages <channel> --kind error --last 10
|
||||
|
||||
# Raw (diagnostic view) — one JSON per line
|
||||
trellis channel messages <channel> --raw --kind progress --last 20
|
||||
trellis channel messages <channel> --raw --last 50
|
||||
```
|
||||
|
||||
Rule of thumb: never diagnose a worker from a truncated progress line.
|
||||
|
||||
### Rebuild Streaming Text
|
||||
|
||||
To reconstruct what a model actually streamed during a turn, concatenate
|
||||
`detail.text_delta` from progress events:
|
||||
|
||||
```bash
|
||||
trellis channel messages <channel> --raw --kind progress --last 80 \
|
||||
| python -c 'import json,sys; [print((json.loads(l).get("detail") or {}).get("text_delta",""), end="") for l in sys.stdin if l.strip()]'
|
||||
```
|
||||
|
||||
## Stalled Worker Diagnosis
|
||||
|
||||
Symptom: `trellis channel list` shows the worker as running, but no new
|
||||
events appear in `messages` and `wait` keeps timing out.
|
||||
|
||||
Triage order:
|
||||
|
||||
1. **Locate the channel files.** Use `list --all --all-projects` if you are
|
||||
not sure which bucket the channel lives in.
|
||||
|
||||
```bash
|
||||
trellis channel list --all --all-projects
|
||||
CHAN=~/.trellis/channels/<bucket>/<channel>
|
||||
```
|
||||
|
||||
2. **Confirm the supervisor and worker PIDs are alive.**
|
||||
|
||||
```bash
|
||||
cat "$CHAN/<worker>.pid" # supervisor PID
|
||||
cat "$CHAN/<worker>.worker-pid" # actual CLI subprocess PID
|
||||
ps -p "$(cat "$CHAN/<worker>.pid")"
|
||||
ps -p "$(cat "$CHAN/<worker>.worker-pid")"
|
||||
```
|
||||
|
||||
If the supervisor PID is gone but the channel still lists the worker,
|
||||
you have a ghost entry — clean it with
|
||||
`trellis channel kill <name> --as <worker> --force`.
|
||||
|
||||
3. **Tail the worker log.** This is the canonical place to see provider /
|
||||
MCP / tool startup output that never makes it onto the channel.
|
||||
|
||||
```bash
|
||||
tail -f "$CHAN/<worker>.log"
|
||||
```
|
||||
|
||||
4. **Check the last raw events.** A worker that emitted `progress` but no
|
||||
`message`/`done` is usually mid-stream or blocked on a tool call:
|
||||
|
||||
```bash
|
||||
trellis channel messages <channel> --raw --last 50
|
||||
```
|
||||
|
||||
Common "alive but silent" causes:
|
||||
|
||||
- Provider cold start before the first token (long, but eventually moves).
|
||||
- A blocking MCP server during startup — visible in the worker log.
|
||||
- Worker is waiting for a tool result whose subprocess hung.
|
||||
- Prompt is huge / model is rate-limited; check provider-side errors in the
|
||||
worker log.
|
||||
|
||||
## Progress Event Interpretation
|
||||
|
||||
A `progress` event represents an in-flight piece of work. Its shape varies
|
||||
by `action` field, but the load-bearing fields are always under `detail`:
|
||||
|
||||
- `detail.text_delta` — incremental model output (concatenate across events
|
||||
to rebuild the streamed reply).
|
||||
- `detail.tool_name`, `detail.tool_input` — tool call about to run or
|
||||
currently running.
|
||||
- `detail.status` — short string used by long-running actions
|
||||
(`starting`, `running`, `flushing`, `done`).
|
||||
- `detail.action` — semantic label (e.g. `status` for thread heartbeats).
|
||||
|
||||
Progress events are **noisy** by design. `wait` ignores them unless you
|
||||
pass `--include-progress`. When you do want to see them, prefer:
|
||||
|
||||
```bash
|
||||
trellis channel messages <channel> --raw --kind progress --last 80
|
||||
```
|
||||
|
||||
A stream that emits progress at a steady cadence but never closes with
|
||||
`done`/`error`/`message` is the classic shape of a hung tool call —
|
||||
inspect the worker log for the subprocess.
|
||||
|
||||
## Wait Semantics (Quick Reference)
|
||||
|
||||
`channel wait` watches `events.jsonl` from EOF and wakes on:
|
||||
|
||||
- `message`
|
||||
- `done`
|
||||
- `error`
|
||||
- `killed`
|
||||
- `progress` only with `--include-progress`
|
||||
|
||||
Useful filters:
|
||||
|
||||
```bash
|
||||
trellis channel wait T --as main --from check --kind done --timeout 15m
|
||||
trellis channel wait T --as main --from check,check-cx --kind done --all --timeout 15m
|
||||
trellis channel wait T --as worker --tag interrupt --timeout 1h
|
||||
trellis channel wait T --as main --thread release-note --action status --timeout 10m
|
||||
```
|
||||
|
||||
Exit codes: `0` matched, `124` timeout, `1`/`2` errors. On `wait --all`
|
||||
timeout, stderr names the workers still missing.
|
||||
|
||||
## Auditing `events.jsonl` — Use Subcommands, Not `grep`
|
||||
|
||||
Every channel persists its full history at `$CHAN/events.jsonl`. It is
|
||||
tempting to `tail` / `grep` / `jq` this file directly during debugging.
|
||||
Don't make it a habit, and **never** do it for forum channels.
|
||||
|
||||
Why subcommands first:
|
||||
|
||||
- `messages` already replays the file with filters (`--kind`, `--from`,
|
||||
`--last`, `--tag`, `--thread`, `--action`) and gives you `--raw` for the
|
||||
exact JSON. Anything you would write a one-liner for, `messages` already
|
||||
does.
|
||||
- `wait` consumes the same file with EOF semantics — re-implementing that
|
||||
with `tail -f | jq` will drop events under load and misorder them under
|
||||
rotation.
|
||||
- `context` materializes a worker's inbox view, including cursor state.
|
||||
Hand-rolled filters do not respect `<worker>.inbox-cursor`.
|
||||
|
||||
### Forum channels: never parse `events.jsonl` directly
|
||||
|
||||
Forum channels multiplex many logical threads onto a single `events.jsonl`.
|
||||
Each event carries `thread`, `action`, and tag fields that the forum
|
||||
subcommands know how to fold together. Parsing the file by hand will:
|
||||
|
||||
- Mix threads together and make a thread look incoherent.
|
||||
- Miss thread lifecycle events (open / status / close) that change how
|
||||
later events should be interpreted.
|
||||
- Ignore worker inbox cursors, so you will "see" events a worker has
|
||||
already consumed and assume they are pending.
|
||||
|
||||
Use the forum-aware views instead:
|
||||
|
||||
```bash
|
||||
# List logical threads inside the forum channel
|
||||
trellis channel forum list <channel>
|
||||
|
||||
# Inspect one thread end-to-end
|
||||
trellis channel thread show <channel> <thread>
|
||||
|
||||
# Replay messages for a thread (supports --raw, --kind, --last)
|
||||
trellis channel messages <channel> --thread <thread> --raw --last 100
|
||||
|
||||
# What a specific worker still has pending
|
||||
trellis channel context <channel> --as <worker>
|
||||
```
|
||||
|
||||
Direct reads of `events.jsonl` are reserved for the case where the CLI
|
||||
itself is suspect — e.g. confirming an event was actually persisted, or
|
||||
diffing against `<worker>.inbox-cursor` while debugging the supervisor.
|
||||
|
||||
## Common Failures
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `trellis: command not found` | CLI not installed globally | `npm install -g @mindfoldhq/trellis` |
|
||||
| `wait` exits immediately | wrong filter or identity collision | use distinct `--as`, inspect raw messages |
|
||||
| zsh errors on message text | shell interpreted punctuation | use `--stdin` or `--text-file` |
|
||||
| progress line is cut off | pretty output truncation | use `messages --raw --kind progress` |
|
||||
| worker never speaks | provider startup / prompt / MCP delay | inspect `<worker>.log`, `ps`, raw events |
|
||||
| channel not found in another cwd | project bucket mismatch | `cd` to project, use `--scope global`, or `list --all-projects` |
|
||||
| ghost worker in list | supervisor died without cleanup | `trellis channel kill <name> --as <worker> --force` |
|
||||
| forum thread looks scrambled | parsed `events.jsonl` directly | use `forum`, `thread`, `messages --thread` |
|
||||
|
||||
## Storage Layout
|
||||
|
||||
```text
|
||||
~/.trellis/channels/
|
||||
└── <bucket>/
|
||||
└── <channel-name>/
|
||||
├── events.jsonl
|
||||
├── <channel>.lock
|
||||
├── <worker>.log
|
||||
├── <worker>.pid
|
||||
├── <worker>.worker-pid
|
||||
├── <worker>.config
|
||||
├── <worker>.session-id
|
||||
├── <worker>.thread-id
|
||||
├── <worker>.inbox-cursor
|
||||
└── <worker>.spawnlock
|
||||
```
|
||||
|
||||
Agents normally use the CLI, not direct file reads. Direct file reads are
|
||||
for debugging when CLI views are insufficient — and even then, never on a
|
||||
forum channel's `events.jsonl`.
|
||||
Reference in New Issue
Block a user