JayPokale/Chisle

In Claude Code, a failed tool call goes to PostToolUseFailure, which cannot rewrite output, so the compressor never sees a failing command. That was checked on Claude Code 2.1.285; see the README's input-axis section. OpenCode's plugin compresses in tool.execute.after (.opencode/plugins/chisle.mjs), and it is unverified whether failed calls reach that hook, and whether rewriting them works. Task: check with a real OpenCode run (a bash command that exits non-zero with >8k chars of output) and document the result in docs/agent-portability.md, with a test if behaviour changes. Done when: the docs state the OpenCode behaviour with the version tested.

The 2026-06-29 Sonnet cross-check (benchmarks/results/2026-06-29-sonnet-cross-check.md) produced the raw-sonnet/ cells, but its prompts were never committed, so those nine tasks cannot be re-run. Only one prompt is quoted in the writeup. Task: reconstruct a runnable Sonnet suite: either recover the original prompts (the answers in raw-sonnet/*.json usually make the ask clear) or write a new, clearly labelled set, as a SUITE=sonnet option in run-live.sh. Done when: SUITE=sonnet bash benchmarks/run-live.sh claude-sonnet-4-6 <fresh-dir> runs end to end, and the writeup links the prompts. You don't need to run it.

The site's "Same prompt. Same model. Real transcripts." section (site/src/components/Compare.tsx) shows June 2026 transcripts from benchmarks/results/raw/, via scripts/build-samples.js. The current numbers come from the 2026-10-01 re-run (benchmarks/results/raw-oct/), and the June cache cell it features is an outlier. Task: let build-samples.js take the transcripts from the October cells (one seed, same six prompts), update the suite label, and keep npm run check:samples passing. Done when: the site shows October transcripts with their real token counts and no hand-edited numbers.

benchmarks/agentic/score.js reports pass rate, files, LOC, deps, output and context tokens, and cost, but not turns per run, even though every cell records turns. Turn count turned out to be the biggest driver of agent-loop cost (see the 2026-10-01 notes in benchmarks/agentic/README.md), and that README's table currently adds it by hand. Task: add a turns column to the Overall table. Default output for existing cells must stay otherwise unchanged. Done when: node benchmarks/agentic/score.js prints turns per arm, and the README table matches the script's output.

benchmarks/run-live.sh redirects each call into "$out" and skips any cell whose file already exists (line ~102). A call that fails or times out still creates the file (empty or partial), so re-running the script silently skips it as "cached" instead of retrying it. Task: write to a temp file and move it into place only on success (both the Claude and Pi branches), so a failed cell stays missing and is retried on the next run. Done when: a killed or failing call leaves no <task>__<arm>.json, a re-run retries it, and cached successful cells are still skipped.