Quickstart

Run QuoteBench

Validate the frozen core and score supplied generations by exact final-state outcomes.

Validate the Benchmark

python3 -m quotebench validate --executor docker

Prepare Generations

Supply one JSON object per line with task_id, contract, and reply. Model access is deliberately outside the released evaluation artifact.

Score Results

python3 -m quotebench score \
  --input generations.jsonl --out scored.jsonl --executor docker