LLM Ground

Consistency · v1

Give the same answer when the input is reordered

One incident list presented twice in different orders, with the same question asked of each. Tests whether position in the input changes the answer — the failure that makes results move when nothing about the data did.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You analyse incident data. When given an output format, follow it exactly — no preamble, no explanation, no extra lines.

user

Below are two lists. They contain the same six incidents in different orders.

Severity ranks SEV1 (highest) above SEV2, SEV2 above SEV3, SEV3 above SEV4.

LIST A
  INC-4185  SEV3  payment webhook retries backing up
  INC-4402  SEV2  search latency above threshold in eu-west
  INC-4471  SEV1  checkout returning 500 for card payments
  INC-4310  SEV4  stale avatar images on profile pages
  INC-4266  SEV2  nightly export finished late
  INC-4498  SEV3  admin audit log missing entries

LIST B
  INC-4310  SEV4  stale avatar images on profile pages
  INC-4498  SEV3  admin audit log missing entries
  INC-4185  SEV3  payment webhook retries backing up
  INC-4266  SEV2  nightly export finished late
  INC-4402  SEV2  search latency above threshold in eu-west
  INC-4471  SEV1  checkout returning 500 for card payments

For each list, identify the single highest-severity incident.

Reply with exactly two lines and nothing else, in this format:

A: <incident id>
B: <incident id>

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Exact match on the two-line answer. Both lists contain INC-4471 at SEV1, so both lines must name it. A model whose answer changes with position produces two different ids, and the severity rule is stated explicitly so no such disagreement can be defended as judgement.

  • Identifies INC-4471 in list A, where the SEV1 sits mid-list
  • Identifies INC-4471 in list B, where it sits last behind two SEV2 entries
  • The two answers agree
  • Exactly two lines, in the given format, with no preamble or explanation

Scoring

Scored by exact match. Deterministic — the same output always produces the same score.

Expected output — surrounding whitespace ignored

A: INC-4471
B: INC-4471

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

A: INC-4471
B: INC-4471

Rejected — answer moved with position — the failure being measured

A: INC-4471
B: INC-4402

Rejected — took the last row of each list

A: INC-4498
B: INC-4471

Rejected — correct, then explained itself

A: INC-4471
B: INC-4471

Both lists contain the same incidents, so the highest severity is INC-4471 in each case.

Rejected — added a preamble

Here are the results:

A: INC-4471
B: INC-4471

Rejected — included the severity alongside the id

A: INC-4471 (SEV1)
B: INC-4471 (SEV1)

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v1 · current
Initial version.

Parameters

maxTokens
1200
temperature
0

Run it yourself

The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You analyse incident data. When given an output format, follow it exactly — no preamble, no explanation, no extra lines."
    },
    {
      "role": "user",
      "content": "Below are two lists. They contain the same six incidents in different orders.\n\nSeverity ranks SEV1 (highest) above SEV2, SEV2 above SEV3, SEV3 above SEV4.\n\nLIST A\n  INC-4185  SEV3  payment webhook retries backing up\n  INC-4402  SEV2  search latency above threshold in eu-west\n  INC-4471  SEV1  checkout returning 500 for card payments\n  INC-4310  SEV4  stale avatar images on profile pages\n  INC-4266  SEV2  nightly export finished late\n  INC-4498  SEV3  admin audit log missing entries\n\nLIST B\n  INC-4310  SEV4  stale avatar images on profile pages\n  INC-4498  SEV3  admin audit log missing entries\n  INC-4185  SEV3  payment webhook retries backing up\n  INC-4266  SEV2  nightly export finished late\n  INC-4402  SEV2  search latency above threshold in eu-west\n  INC-4471  SEV1  checkout returning 500 for card payments\n\nFor each list, identify the single highest-severity incident.\n\nReply with exactly two lines and nothing else, in this format:\n\nA: <incident id>\nB: <incident id>"
    }
  ],
  "max_tokens": 1200,
  "temperature": 0
}'