Run QuoteBench

1. Validate the frozen core

Every task's oracle command must pass its exact-final-state validator before anything else runs.

python3 -m quotebench validate --executor docker

2. Run your own model

Point the runner at any OpenAI-compatible chat-completions endpoint. Run the raw contract and the disclosed-boundary contract; each run generates one command per task, executes it on its own transport, and scores the final state. Expect 56 calls per contract per trial.

export QUOTEBENCH_API_KEY=...
python3 -m quotebench run --model MODEL_ID \
  --base-url https://api.example.com/v1 \
  --contract raw --out raw.jsonl --executor docker
python3 -m quotebench run --model MODEL_ID \
  --base-url https://api.example.com/v1 \
  --contract nested-shell-v2 --out disclosed.jsonl --executor docker

3. Decompose the crossover

The crossover command replays every stored reply through both transports and prints RR, RN, NR, NN, transport damage (RN − RR), contract-conditioned compensation (NN − RN), and the matched gap (NN − RR) for your model.

python3 -m quotebench crossover --raw raw.jsonl --disclosed disclosed.jsonl

The run-your-model guide covers flags, resume, and how to read the four cells.

Score generations you already have

Supply one JSON object per line with task_id, contract, and reply; the scorer executes each reply on the transport its contract implies.

python3 -m quotebench score \
  --input generations.jsonl --out scored.jsonl --executor docker