Dataset card
QuoteBench Datasheet
A programmatically generated, execution-verified benchmark for byte-exact shell quoting and escaping tasks.
Motivation
QuoteBench was created to isolate incorrect shell quoting and escaping when LLM agents drive a shell tool directly. Terminal and agentic benchmarks exercise this failure mode only incidentally; NL-to-command benchmarks often use metrics that erase argument values, where quoting errors live.
Composition
- Instances: 56 frozen-core tasks, with tier-0 benign controls and tier-1..3 hostile variants.
- Fields: fixture, natural-language instruction, state validator, oracle command, and naive probe.
- Hazard tags: ShellCheck-style quoting families and hostile filename classes.
- Splits: evaluation-only; no train/test split in the public core.
Collection and Generation
The benchmark is generated by Python code in quotebench/scenarios.py. Fixtures are built without invoking a shell, so the benchmark construction path does not introduce its own quoting problem. Tokens are deterministic and regeneration is byte-identical.
Intended Uses
- Compare shell-quoting skill across models and harness contracts.
- Study raw, JSON, and wrapped command transport effects.
- Measure pass@1 and repeated-trial reliability for a tail-risk micro-skill.
- Serve as a template for isolating other verifiable skills.
Out of Scope
QuoteBench is not a general shell-competence or end-to-end agent benchmark. Absolute cross-vendor numbers should be read as broad tiers because provider sampling controls differ. Results are toolchain-dependent, so reports should state GNU versus BSD execution.
Distribution
The public repository contains the frozen core, harness, adapters, scorer, replay path, Dockerfile, headline figures, Apache-2.0 license, NOTICE attribution, and CITATION.cff metadata.
Credits
Author list and formal acknowledgements will be updated with the project paper.