Benchmark specification
Task Model, Contracts, Metrics
QuoteBench measures one narrow skill: composing a single correct bash -c payload when exact bytes must survive shell quoting, nested transport, and hostile filenames.
What It Measures
QuoteBench tests whether LLM agents can compose one shell command whose only real difficulty is quoting. The underlying operations are deliberately ordinary: write a file, replace a string, count lines, rename files, pass argv, set environment variables, or commit an exact message.
The headline metric is the Quoting Gap: benign tier-0 pass rate minus hostile tier-1..3 pass rate within the same scenario. Because the operation is identical, the drop attributes failure to quoting rather than tool knowledge.
Task Anatomy
- Fixture: created by Python filesystem calls, not shell scripts.
- Instruction: natural language with exact literal payloads delimited by
⟪...⟫. - Validator: checks final file bytes, argv received by a probe, parsed JSON, or git state.
- Oracle: machine-constructed known-good command; validation requires oracle 56/56.
- Naive probe: tempting wrong quoting that must pass benign controls and fail hostile raw tasks.
- Environment policy: extra non-whitelisted files are
environment-invalidand receive zero credit.
Scenario Matrix
| Scenario | Operation | Hazard axis |
|---|---|---|
| write-file | Create exact file content | Quote choice, dollar and backtick content |
| sed-replace | Literal string replacement | Shell quoting plus sed escaping |
| grep-count | Count exact-substring lines | Regex-vs-literal, dollar expansion |
| field-lookup | awk column lookup | $1 shell-vs-awk collision |
| hostile-filenames | Rename or delete one file | Spaces, glob-lookalikes, leading dashes |
| find-glob | List matching files | Unquoted patterns and cwd trap files |
| ssh-nested | Create file through ssh-like wrapper | Double shell evaluation and client-side expansion |
| json-write | Write JSON exact value | JSON escaping composed with shell quoting |
| git-commit | Commit exact message | Apostrophes, multiline messages, command substitution |
| env-passing / argv-passing | Pass exact env values or argv strings | Assignment quoting, word splitting, glob expansion |
| bulk-rename | Prefix all matching files | Loop quoting with hostile names |
| heredoc-write / ssh-heredoc | Create exact multiline files | Delimiter expansion, nested heredocs, delimiter collision |
Tool Contracts
- raw: reply is executed verbatim through
bash -c. - json: reply must be strict JSON with a
commandfield. - wrapped: reply is interpolated into another
bash -c "..."layer, modeling harness transport.
Under json and wrapped, controls can carry intrinsic transport difficulty. Contract deltas are therefore the clean comparison there; the attribution-clean Quoting Gap is the raw-contract one.
Validity Checks
- Oracle passes all 56 tasks.
- Naive command passes zero hostile raw tasks.
- Raw controls remain easy, so quoting is the measured difficulty.
Run python3 -m quotebench validate before trusting any number.