QuoteBench — How Matched Scores Can Hide Command-Path Failures

What the crossover shows

Each task asks for one Bash command and is scored by exact final state, so any correct quoting strategy passes. Eight same-window configurations supply the controlled evidence.

Transport damage

The same command loses 55–73 points behind one added parser.

Replaying each raw reply through one added bash -c layer lowers exact-final-state success by 55.4–73.2 points in all eight configurations. A real ssh boundary reproduces the damage for seven of eight to the decimal.

Compensation

Seeing the boundary recovers 30–61 points for six configurations.

Replies generated under a disclosed-boundary contract recover 30.4–60.7 points on the nested path for six configurations and zero or slightly negative for the other two. Two further public draws keep every damage negative and change no compensation sign.

Matched scores

A matched score can hide the dependency.

The two effects cancel at one frontier configuration: GPT-5.6-sol’s matched gap of −3.6 hides −64.3 damage and +60.7 compensation. Elsewhere the matched score falls with the path, and the deployment configuration reorders models (Kendall τ 0.57 between raw and nested orderings).

Attribution

A known hazard, attributed to its cause.

Quoting failures are documented; QuoteBench holds the reply fixed and changes one parser, so each loss is attributed to transport or to generation. A fixed-commit survey of six agent systems finds boundaries applied downstream of the stated contract, such as Codex executing a native command field as shell -c. Correct escaping or a temporary script removes the effect, and evaluations should report contract, execution path, operating point, and validator.

Results

The same-window crossover is the controlled evidence. The scorecards below it are descriptive summaries of the released result files.

Study A — same-window crossover (controlled)

model	RR	RN	NR	NN	damage	compensation	matched gap
GPT-5.6-sol	94.6	30.4	55.4	91.1	−64.3	+60.7	−3.6
GPT-5.5	100.0	28.6	50.0	89.3	−71.4	+60.7	−10.7
Opus-5	96.4	30.4	42.9	89.3	−66.1	+58.9	−7.1
Gemini-3.1-Pro	98.2	25.0	33.9	80.4	−73.2	+55.4	−17.9
Gemini-3.5-Flash	96.4	28.6	67.9	58.9	−67.9	+30.4	−37.5
Opus-4.8	91.1	26.8	62.5	57.1	−64.3	+30.4	−33.9
Qwen3.5-27B	85.7	30.4	83.9	30.4	−55.4	0.0	−55.4
Gemini-3.1-Flash-Lite	78.6	19.6	80.4	14.3	−58.9	−5.4	−64.3

Trial 0, GNU replay, percentages of 56 tasks. RR/RN: raw reply on the raw/nested path; NR/NN: disclosed-boundary reply on the raw/nested path. Damage = RN − RR, compensation = NN − RN, matched gap = NN − RR.

Study A — best-observed scorecard (descriptive)

model	best observed setting	control	hostile	all 56	hostile LOFO
GPT-5.5	xhigh	14/14	42/42	100.0%	[100.0, 100.0]
Opus-5	xhigh	14/14	42/42	100.0%	[100.0, 100.0]
Fable-5	max	14/14	42/42	100.0%	[100.0, 100.0]
GPT-5.6-sol	high	14/14	41/42	98.2%	[97.4, 100.0]
Opus-4.8	max	12/14	41/42	94.6%	[97.4, 100.0]
Gemini-3.1-Pro	low	14/14	37/42	91.1%	[87.2, 89.7]
Sonnet-4.6	high	9/14	27/42	64.3%	[61.5, 69.2]
Gemini-3.5-Flash	medium	10/14	26/42	64.3%	[59.0, 66.7]
Haiku-4.5	medium	8/14	13/42	37.5%	[28.2, 33.3]
Qwen3.5-27B	non-think	8/14	9/42	30.4%	[15.4, 23.1]
Qwen3.5-4B	think	5/14	7/42	21.4%	[10.3, 17.9]
Qwen3.5-9B	non-think	5/14	5/42	17.9%	[5.1, 12.8]
Gemini-3.1-Flash-Lite	default	6/14	2/42	14.3%	[2.6, 5.1]

Trial-0 matched-nested final-state success on the same 56-task core in the pinned GNU/Linux replay. Updated 2026-08-05. Each base model contributes its highest-scoring measured setting; ties prefer default, then lower reported output tokens. This is a descriptive best-observed summary.

Study B — provider-native shell-tool campaign (observational)

model configuration	paired task–rung trials	raw	native	Δ
Opus-4.8	840	95.4%	98.0%	+2.6
Opus-5	840	98.2%	97.4%	-0.8
Fable-5	840	99.3%	97.1%	-2.1
Gemini-3.1-Pro	504	98.8%	95.0%	-3.8
GPT-5.6-sol	672	96.9%	94.3%	-2.5
Gemini-3.5-Flash	672	95.7%	85.7%	-10.0

GNU replay, averaged over every measured effort rung and three trials per task/protocol cell. Updated 2026-08-05. This campaign measures a structured shell-tool boundary and is not numerically commensurable with the best-observed Study-A scorecard.

Headline Charts

Study A on the reproducible GNU/Linux toolchain; BSD companion numbers and the Study B figures are in the paper.

Matched nested success at each exposed effort setting, by model group — **Study A: effort ladders.**Matched nested success at each exposed effort setting. Ladders are non-monotonic, and the unset default lands on a different rung per provider.

Crossover cells at every measured effort rung for six configurations — **Study A: crossover at every rung.**Damage stays negative and compensation keeps its sign along each ladder; trial-0 ladders move nested-replay pass rates by at most 5.4 points.

Run your own model

The public package generates, executes, and scores with the Python standard library and Docker. Point it at any OpenAI-compatible endpoint, run the two contracts, and the crossover command prints RR, RN, NR, NN, damage, and compensation for your model. The guide has the details and the cost per run.

export QUOTEBENCH_API_KEY=...
python3 -m quotebench run --model MODEL_ID \
  --base-url https://api.example.com/v1 \
  --contract raw --out raw.jsonl --executor docker
python3 -m quotebench run --model MODEL_ID \
  --base-url https://api.example.com/v1 \
  --contract nested-shell-v2 --out disclosed.jsonl --executor docker
python3 -m quotebench crossover --raw raw.jsonl --disclosed disclosed.jsonl

Project Files

Start from the spec if you want the benchmark design, or the datasheet if you need release metadata.

QuickstartCommands for validation and scoring supplied generations. SpecificationTask model, contracts, metrics, and validity checks. Related WorkWhy quoting is not isolated by existing terminal benchmarks. CitationCitation metadata and attribution guidance.