Bake-off: a prompt set through the room

Send your own prompts, optionally with the answers you expect, and rival frontier models answer each one blind, then cross-examine each other. You get a two-page bake-off report: where the seats agreed, where the agreement held under challenge, and, when you supplied expected answers, where the room agreed and was still wrong. Your own model can sit in the room too.

HTTPPOST https://collider.run/api/report{ "mode": "bakeoff", … }MCPstart_bakeoff get_report

What happens to each item

Blind round. Every seat answers your prompt on its own, without seeing any other seat. Each turn must end with a FINAL: line (one sentence) and a CONFIDENCE: line. This blind answer is what the model would have told your users alone.
Cross-examination. The seats see each other’s answers, challenge or endorse, and restate their FINAL line. A seat may hold, move, or concede to the majority.
Digest. Agreement is measured over the blind answers only. When you supplied an expected answer, each seat’s blind answer is graded against it; a seat that changed its answer has the changed answer graded separately.

Seats without a web tool are told in the brief that they have none; a seat is never told it can search when it cannot. expected and rubric are never shown to a seat.

Request

POST /api/report · Authorization: Bearer clsk_…

{
  "mode": "bakeoff",
  "title": "Q3 model choice",
  "audience": "our support agents",
  "items": [
    { "id": "capital", "prompt": "What is the capital of Australia? City name only.", "expected": "Canberra" },
    { "id": "refund",  "prompt": "A customer paid twice by mistake. What do we do?",
      "context": "Policy: duplicate charges are refunded in full within 5 working days.",
      "expected": "Refund the duplicate charge in full within 5 working days." }
  ],
  "rounds": 2,
  "responseLength": "standard",
  "webGrounding": false,
  "lineup": ["chatgpt", "claude", "gemini", "mistral"],
  "notifyEmail": "[email protected]"
}

Field	Required	Default	Notes
`title`, `audience`	yes		Max 200 chars each; printed on the sheet, the audience shapes the brief.
`items[]`	yes		1 to 40. `prompt` ≤ 8,000 chars; `context` ≤ 20,000 (given to the seats); `expected`, `rubric` ≤ 2,000 (grading only); `id` optional, `item-N` otherwise.
`rounds`	no	2	2 or 3. Round 1 is blind.
`responseLength`	no	standard	`brief` · `standard` · `detailed` · `review`.
`webGrounding`	no	false	Seats with a web tool may use it; the rest are told they have none.
`lineup`	no	active lineup (+ custom seats)	2 to 5 of `chatgpt` · `gemini` · `claude` · `grok` · `mistral` · `custom1..4`.
`models`	no	active builds	Per-seat model pin for registry seats, vendor-family allowlisted.
`customSeats`	no	none	See below.
`notifyEmail`, `notify`	no	account, true	One email when the run finishes; `notify: false` for none.

Accepted: 202 { reportId, cellsTotal, credits: { charged, breakdown }, runtime: { models, rounds, … } }. Rejected before anything is charged: over 40 items, a lineup outside 2–5, another run already in progress on the account (409), insufficient credits (402, with the estimate), a custom seat that fails its preflight (400).

Polling and the result

GET /api/report/{reportId}                  → { status, stage, cellsComplete, cellsTotal, … }
GET /api/report/{reportId}                  → on completion: { findings: <bundle>, pdf, html, … }
GET /api/report/{reportId}?format=pdf       → the bake-off report PDF
GET /api/report/{reportId}?format=html      → the same, as HTML
DELETE /api/report/{reportId}               → erase everything held for the run

A run takes tens of minutes to a couple of hours (one collision per item, one at a time). When it finishes, the account (or notifyEmail) gets one email with a link to the signed-in page /reports/{reportId}, which serves the PDF and offers the same erase.

The bake-off report

Page 1: summary counters, the per-seat table, and the items to read first. Page 2: every item. Then the method box.

Field	Meaning
Split	Items with no majority blind answer.
Brittle	Items whose verdict heuristic read the agreement as not having been tested by challenge. A flag for where to look, not a grade.
Agreed & wrong	Items where the majority blind answer decisively mismatched your expected answer. An agreed answer the machine could not grade is reported as “ungraded”, never as wrong.
Blind agree w/ majority	Per seat: of items with a majority, how often this seat’s blind answer was in it.
Held after challenge	Per seat: how often its final answer was the same claim as its blind answer (paraphrases count as held).
Correct, blind	Per seat, with expected answers: share of decisively graded blind answers that matched, with the counts.
Right→wrong · wrong→right	Per seat: items where the seat held a correct answer blind and gave it up under challenge, and items where it was wrong alone and the room corrected it.
Per item	Agreement class, verdict label, each seat’s blind FINAL → last FINAL with its mark, your expected answer, agreed-vs-expected, flags.

What the marks mean, and what they cannot tell you

Grading ladder. Two answers are matched cheapest and safest first: exact, then normalised (case, punctuation, markdown), then numeric within a small tolerance, and only then by a language-model equivalence call. Two clean numbers that differ never reach the model.

Every machine judgement is flagged. A match or mismatch the model decided carries grader; unless the model was highly confident it also carries needs a human look. A non-match is only called incorrect when the machine could actually tell (clean differing numbers, or a highly confident grader); otherwise it is left ungraded and is not counted as wrong anywhere.

No mark without the evidence. A correctness mark is never printed without the seat’s actual FINAL line and your expected answer on the same page, so you can judge every call yourself. The model that made any equivalence call is named in the method box.

The verdict label is a heuristic. Earned / brittle / contested comes from an unvalidated v0 reading of certainty spread, carried verbatim and labelled as such. On our own kill test it misread the majority of the debates it labelled. It is a pointer to where to look, never a grade.

What this cannot tell you: whether an answer is right in the world. It measures what independently built models answered, where they agreed, and whether the agreement survived challenge. Agreement is evidence, not proof.

Why the flags: in our registered kill test the instrument’s own confident classifier was wrong on 47% of the hard cases. The provenance flags, the “needs a human look” flag and the “ungraded, not wrong” rule exist so that a number on this sheet never hides a guess.

Custom seats: your own model in the room

Any OpenAI-compatible chat endpoint (a hosted open-weights model, a vLLM box, your own key with a provider) can take a seat next to the registry seats, addressed by the label you give it.

"customSeats": [{
  "key": "custom1",
  "label": "llama-70b",
  "baseUrl": "https://inference.example.com/v1",
  "model": "meta-llama/Llama-3.3-70B",
  "apiKey": "…",
  "priceIn": 0.2, "priceOut": 0.6
}],
"lineup": ["chatgpt", "claude", "custom1"]

Field	Notes
`key`	`custom1` to `custom4`; at most four custom seats.
`label`	One token, 1–24 letters, digits, dot, underscore or hyphen. How the other seats address it (`@label`). Registry names are reserved.
`baseUrl`	https, port 443, a public host, no credentials or query string. The SDK appends `/chat/completions`, so include `/v1` where the server expects it.
`model`	The model name the endpoint expects.
`apiKey`	Sent only to your endpoint. Checked with one minimal call before anything is charged; held sealed (AES-256-GCM) only while the run needs it; wiped when the run ends, fails, is deleted or purged; never echoed, never logged.
`priceIn`, `priceOut`	Optional, USD per 1M tokens, for the run’s own cost accounting. Absent = “cost unrecorded” in the method box.

In the room the seat gets the same token cap per turn as every other seat, no web tool and no images; it is not given any special treatment in grading. We do not control, log into or audit your endpoint: what it serves is what is cross-examined, and the seat legend prints model @ host so the report says exactly what ran.

Pricing

Per item, plus one synthesis fee per run. The exact breakdown is printed in the accept response before anything runs; the defaults are 60 credits per item and 200 for synthesis, with the same grounding and extra-round surcharges as a report.
Items that did not run are refunded; a run that fails before producing anything is refunded in full.
Personal API keys allow 20 debates per day, so a bake-off over 20 items needs a partner key; one run at a time per account.
The report never shows our provider cost; it shows what ran (items, rounds, seats, builds, grader model, dates).

Retention

The prompt set, every debate transcript and the rendered sheets are held only for the life of the run and its retention window (30 days for the source; artifacts until you erase them). DELETE /api/report/{id}, or the erase button on the report page, removes the prompts, findings, HTML, PDF, per-item digests and the debate transcripts at once. The job row stays as a billing record with no content.

Measurement method and validation published (pre-registered study, report and data on OSF); calibration corpus and prompts are ours.