LLM Ground

Consistency · v1

Normalise five differently-written values into one format

Five US phone numbers in five notations, all to be returned in E.164. Tests whether a model applies one rule uniformly or drifts toward the shape of each input. Normalising four of five produces a column that looks clean and cannot be matched on — worse than one that is visibly messy.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You normalise data. Apply the stated rule identically to every input. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences.

user

Normalise each phone number below to E.164: a leading plus sign, then the country
code, then the digits, with no spaces, brackets, dots or hyphens. All five are US
numbers, country code 1.

  c1  (415) 555-0132
  c2  415.555.0198
  c3  +1 415 555 0143
  c4  4155550176
  c5  1-415-555-0187

Return a JSON object with exactly the keys c1 through c5, each mapping to the
normalised string. Reply with the JSON only.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

All five values are compared exactly. Each input format is a different way to be led astray: brackets, dots, an existing country code, no punctuation at all, and a leading 1 without a plus. The probe fails if any single one is not in E.164, because a partially normalised column is not usable.

  • Parses as JSON with no surrounding prose or code fence
  • Exactly the keys c1 through c5, with no extras
  • Every value begins with +1 and contains only digits after the plus
  • c3 is not double-prefixed despite already carrying its country code
  • c4 gains the country code it was missing
  • c5's leading 1 becomes +1 rather than being kept as a digit or dropped

Scoring

Scored by JSON Schema. Deterministic — the same output always produces the same score.

JSON Schema — the output must be bare JSON, no code fence

{
  "type": "object",
  "required": [
    "c1",
    "c2",
    "c3",
    "c4",
    "c5"
  ],
  "additionalProperties": false,
  "properties": {
    "c1": {
      "const": "+14155550132"
    },
    "c2": {
      "const": "+14155550198"
    },
    "c3": {
      "const": "+14155550143"
    },
    "c4": {
      "const": "+14155550176"
    },
    "c5": {
      "const": "+14155550187"
    }
  }
}

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

{"c1":"+14155550132","c2":"+14155550198","c3":"+14155550143","c4":"+14155550176","c5":"+14155550187"}

Rejected — four normalised, one echoed in its input shape — the drift being measured

{"c1":"+14155550132","c2":"+14155550198","c3":"+1 415 555 0143","c4":"+14155550176","c5":"+14155550187"}

Rejected — double-prefixed the number that already had a country code

{"c1":"+14155550132","c2":"+14155550198","c3":"+1+14155550143","c4":"+14155550176","c5":"+14155550187"}

Rejected — missed the country code on the bare ten-digit input

{"c1":"+14155550132","c2":"+14155550198","c3":"+14155550143","c4":"+4155550176","c5":"+14155550187"}

Rejected — kept the leading 1 as a digit rather than a country code

{"c1":"+14155550132","c2":"+14155550198","c3":"+14155550143","c4":"+14155550176","c5":"+114155550187"}

Rejected — added a sixth key explaining itself

{"c1":"+14155550132","c2":"+14155550198","c3":"+14155550143","c4":"+14155550176","c5":"+14155550187","note":"All numbers normalised to E.164 assuming US country code 1."}

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v1 · current
Initial version.

Parameters

maxTokens
1600
temperature
0

Run it yourself

The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You normalise data. Apply the stated rule identically to every input. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences."
    },
    {
      "role": "user",
      "content": "Normalise each phone number below to E.164: a leading plus sign, then the country\ncode, then the digits, with no spaces, brackets, dots or hyphens. All five are US\nnumbers, country code 1.\n\n  c1  (415) 555-0132\n  c2  415.555.0198\n  c3  +1 415 555 0143\n  c4  4155550176\n  c5  1-415-555-0187\n\nReturn a JSON object with exactly the keys c1 through c5, each mapping to the\nnormalised string. Reply with the JSON only."
    }
  ],
  "max_tokens": 1600,
  "temperature": 0
}'