LLM agent evaluation
An execution-verified benchmark that crosses generation contract with command transport, exposing failures that matched-path scores can hide.
Each task asks for one bash command, scored by final program state, not command text, so any correct quoting strategy passes. Two studies share the 56-task core.
Across all eight same-window configurations, replaying the same raw-conditioned reply through one added parser lowers success by 55.4–73.2 points. Disclosing that boundary before generation produces substantial compensation in six configurations.
Through the structured tool-call API real agents use, the effect ranges from a stable benefit (Opus-4.8) to a stable tax (Gemini-3.5-flash), and flips sign across BSD/GNU for others.
Tool-call adherence stays 98–100% while native accuracy spans 86–100%. Calling the right tool with valid arguments does not mean the command runs correctly.
Which models survive a hostile transport contract. Auto-generated from the released result files on every publish.
| model | best observed setting | control | hostile | all 56 | hostile LOFO |
|---|---|---|---|---|---|
| GPT-5.5 | xhigh | 14/14 | 42/42 | 100.0% | [100.0, 100.0] |
| Opus-5 | xhigh | 14/14 | 42/42 | 100.0% | [100.0, 100.0] |
| Fable-5 | max | 14/14 | 42/42 | 100.0% | [100.0, 100.0] |
| GPT-5.6-sol | high | 14/14 | 41/42 | 98.2% | [97.4, 100.0] |
| Opus-4.8 | max | 12/14 | 41/42 | 94.6% | [97.4, 100.0] |
| Gemini-3.1-Pro | low | 14/14 | 37/42 | 91.1% | [87.2, 89.7] |
| Sonnet-4.6 | high | 9/14 | 27/42 | 64.3% | [61.5, 69.2] |
| Gemini-3.5-Flash | medium | 10/14 | 26/42 | 64.3% | [59.0, 66.7] |
| Haiku-4.5 | medium | 8/14 | 13/42 | 37.5% | [28.2, 33.3] |
| Qwen3.5-27B | non-think | 8/14 | 9/42 | 30.4% | [15.4, 23.1] |
| Qwen3.5-4B | think | 5/14 | 7/42 | 21.4% | [10.3, 17.9] |
| Qwen3.5-9B | non-think | 5/14 | 5/42 | 17.9% | [5.1, 12.8] |
| Gemini-3.1-Flash-Lite | default | 6/14 | 2/42 | 14.3% | [2.6, 5.1] |
Trial-0 matched-nested final-state success on the same 56-task core in the pinned GNU/Linux replay. Updated 2026-08-05. Each base model contributes its highest-scoring measured setting; ties prefer default, then lower reported output tokens. This is a descriptive best-observed summary.
| model configuration | paired task–rung trials | raw | native | Δ |
|---|---|---|---|---|
| Opus-4.8 | 840 | 95.4% | 98.0% | +2.6 |
| Opus-5 | 840 | 98.2% | 97.4% | -0.8 |
| Fable-5 | 840 | 99.3% | 97.1% | -2.1 |
| Gemini-3.1-Pro | 504 | 98.8% | 95.0% | -3.8 |
| GPT-5.6-sol | 672 | 96.9% | 94.3% | -2.5 |
| Gemini-3.5-Flash | 672 | 95.7% | 85.7% | -10.0 |
GNU replay, averaged over every measured effort rung and three trials per task/protocol cell. Updated 2026-08-05. This campaign measures a structured shell-tool boundary and is not numerically commensurable with the best-observed Study-A scorecard.
The two studies, on the reproducible GNU/Linux toolchain (macOS/BSD companion numbers are in the paper).
The core benchmark uses Python standard library only. The public artifact validates tasks and scores supplied generations; model access remains outside the release.
python3 -m quotebench validate --executor docker
python3 -m quotebench score \
--input generations.jsonl --out scored.jsonl --executor dockerStart from the spec if you want the benchmark design, or the datasheet if you need release metadata.