Dataset card

QuoteBench Datasheet

A programmatically generated, execution-verified benchmark for byte-exact shell quoting and escaping tasks.

56 core tasks14 scenario families times 4 hostility tiers.

No personal dataReleased tasks are generic distillations, not private commands.

State-based labelsGround truth is final program state, not a reference string.

Apache-2.0Redistribution retains license, copyright, and NOTICE attribution.

Motivation

QuoteBench was created to isolate incorrect shell quoting and escaping when LLM agents drive a shell tool directly. Terminal and agentic benchmarks exercise this failure mode only incidentally; NL-to-command benchmarks often use metrics that erase argument values, where quoting errors live.

Composition

Instances: 56 frozen-core tasks, with tier-0 benign controls and tier-1..3 hostile variants.
Fields: fixture, natural-language instruction, state validator, oracle command, and naive probe.
Hazard tags: ShellCheck-style quoting families and hostile filename classes.
Splits: evaluation-only; no train/test split in the public core.

Collection and Generation

The benchmark is generated by Python code in quotebench/scenarios.py. Fixtures are built without invoking a shell, so the benchmark construction path does not introduce its own quoting problem. Tokens are deterministic and regeneration is byte-identical.

Intended Uses

Compare shell-quoting skill across models and harness contracts.
Study raw, JSON, and nested-shell command transport effects.
Measure pass@1 and repeated-trial reliability for a tail-risk micro-skill.
Serve as a template for isolating other verifiable skills.

Out of Scope

QuoteBench is not a general shell-competence or end-to-end agent benchmark. Absolute cross-vendor numbers should be read as broad tiers because provider sampling controls differ. Results are toolchain-dependent, so reports should state GNU versus BSD execution.

Distribution

The public repository contains the frozen core, harness, offline validator and scorer, rollout verification and descriptive-analysis commands, Dockerfile, headline figures, Apache-2.0 license, NOTICE attribution, and CITATION.cff metadata.

Credits

QuoteBench was created by Shangao Li, Yao Zhang, Volker Tresp, and Yuanyuan Yang. We gratefully acknowledge Jiafu Tang and Ziyu Zhou for providing access to the Gemini and GPT APIs, respectively.