LLM agent evaluation

QuoteBench

A focused benchmark for the shell quoting and escaping failures that broad coding benchmarks usually hide.

View repository Run the benchmark Read datasheet
Raw vs wrapped pass rates across models
Wrapped harnesses expose hidden quoting failures that raw shell execution can miss.
56 tasksfrozen, execution-verified core
12 systemsfrontier, proprietary, and open-weight models
up to -64 ptsraw-to-wrapped contract collapse
100% oraclesingle-command solvability validation

What QuoteBench Isolates

Each task asks for one bash command. Scoring checks final state, not command text, so models can solve the task however they want as long as they preserve the exact bytes.

Contract

Raw shell skill is not enough.

The same command can fail when a harness wraps it in another quoting layer. QuoteBench measures raw, JSON, and wrapped transport separately.

Effort

Reasoning budget can saturate.

Some models spend more generated tokens without improving accuracy; others convert effort into concrete gains.

Reliability

pass@1 hides tail risk.

Repeated trials expose whether a model is consistently safe on byte-exact shell tasks, not merely lucky once.

Headline Charts

The public repository includes the benchmark core, scorer, replay path, datasheet, and current headline figures.

Accuracy versus generated token cost under wrapped contract
Effort saturation.Accuracy improves only when extra computation is usable for the quoting task.
Qwen thinking toggle by size
Thinking toggle.Mid-capability models gain the most.
pass@1 versus pass^k reliability
Reliability.pass@1 can overstate tail performance.

Run It Locally

The core benchmark uses Python standard library only. Use the Docker executor for untrusted model output and pinned GNU userland.

python3 -m quotebench validate
OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=... \
  python3 -m quotebench run --adapter openai --model MODEL \
  --contract wrapped --executor docker --out results/run.jsonl
python3 -m quotebench score results/run.jsonl

Project Files

Start from the spec if you want the benchmark design, or the datasheet if you need release metadata.

QuickstartCommands for validation, model runs, and scoring. SpecificationTask model, contracts, metrics, and validity checks. Related WorkWhy quoting is not isolated by existing terminal benchmarks. CitationCitation metadata and attribution guidance.