Quickstart

Run QuoteBench

Validate the frozen core, run a model through an API-compatible adapter, and score pass rates, failure taxonomy, and token usage.

Validate the Benchmark

python3 -m quotebench list
python3 -m quotebench show ssh-heredoc/t3-gnarly
python3 -m quotebench validate

Run a Model

OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=... \
  python3 -m quotebench run --adapter openai --model MODEL \
  --contract wrapped --executor docker --out results/run.jsonl

Use --executor docker for untrusted model output and pinned GNU userland.

Score Results

python3 -m quotebench score results/run.jsonl
QB_PRICE_IN=1.25 QB_PRICE_OUT=10 python3 -m quotebench score results/run.jsonl