Consistency · v2

Hold a schema steady across ten generated items

Generate ten records against a fixed schema in one response. Format drift usually appears late in a list — item ten gains a field, or an id becomes a string — so every item is validated rather than sampled.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v2 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You generate test fixtures. Reply with a single JSON array and nothing else. Every object must have exactly the same shape.

user

Generate exactly 10 fictional user records as a JSON array.

Each object must have exactly these keys and types:
  id      integer, 1 through 10, in order
  name    non-empty string
  active  boolean

No other keys. No nulls. Reply with the array only.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Every one of the ten objects is validated against the schema — drift in a single late item fails the probe, which is exactly the behaviour being measured.

Parses as JSON with no prose or code fence
Exactly ten objects
Every object has exactly id, name and active — no extras, none missing
id is an integer in every object, never a string
active is a boolean in every object
No null values anywhere

Scoring

Scored by JSON Schema. Deterministic — the same output always produces the same score.

JSON Schema — the output must be bare JSON, no code fence

{
  "type": "array",
  "minItems": 10,
  "maxItems": 10,
  "items": {
    "type": "object",
    "required": [
      "id",
      "name",
      "active"
    ],
    "additionalProperties": false,
    "properties": {
      "id": {
        "type": "integer"
      },
      "name": {
        "type": "string",
        "minLength": 1
      },
      "active": {
        "type": "boolean"
      }
    }
  }
}

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

[{"id":1,"name":"User 1","active":true},{"id":2,"name":"User 2","active":false},{"id":3,"name":"User 3","active":true},{"id":4,"name":"User 4","active":false},{"id":5,"name":"User 5","active":true},{"id":6,"name":"User 6","active":false},{"id":7,"name":"User 7","active":true},{"id":8,"name":"User 8","active":false},{"id":9,"name":"User 9","active":true},{"id":10,"name":"User 10","active":false}]

Rejected — drift in the last item — an extra field

[{"id":1,"name":"User 1","active":true},{"id":2,"name":"User 2","active":true},{"id":3,"name":"User 3","active":true},{"id":4,"name":"User 4","active":true},{"id":5,"name":"User 5","active":true},{"id":6,"name":"User 6","active":true},{"id":7,"name":"User 7","active":true},{"id":8,"name":"User 8","active":true},{"id":9,"name":"User 9","active":true},{"id":10,"name":"User 10","active":true,"role":"admin"}]

Rejected — one id became a string

[{"id":1,"name":"User 1","active":true},{"id":2,"name":"User 2","active":true},{"id":3,"name":"User 3","active":true},{"id":4,"name":"User 4","active":true},{"id":5,"name":"User 5","active":true},{"id":6,"name":"User 6","active":true},{"id":7,"name":"User 7","active":true},{"id":8,"name":"User 8","active":true},{"id":9,"name":"User 9","active":true},{"id":"10","name":"User 10","active":true}]

Rejected — nine items instead of ten

[{"id":1,"name":"User 1","active":true},{"id":2,"name":"User 2","active":true},{"id":3,"name":"User 3","active":true},{"id":4,"name":"User 4","active":true},{"id":5,"name":"User 5","active":true},{"id":6,"name":"User 6","active":true},{"id":7,"name":"User 7","active":true},{"id":8,"name":"User 8","active":true},{"id":9,"name":"User 9","active":true}]

Rejected — a null slipped in

[{"id":1,"name":"User 1","active":true},{"id":2,"name":"User 2","active":true},{"id":3,"name":"User 3","active":true},{"id":4,"name":"User 4","active":true},{"id":5,"name":"User 5","active":true},{"id":6,"name":"User 6","active":true},{"id":7,"name":"User 7","active":true},{"id":8,"name":"User 8","active":true},{"id":9,"name":"User 9","active":true},{"id":10,"name":null,"active":true}]

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v2 · current: Initial version. Measures intra-response consistency; run-to-run consistency needs multi-sample execution support and is tracked separately.

Parameters

maxTokens: 1700
temperature: 0

Run it yourself

The probe exactly as it stands at v2. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You generate test fixtures. Reply with a single JSON array and nothing else. Every object must have exactly the same shape."
    },
    {
      "role": "user",
      "content": "Generate exactly 10 fictional user records as a JSON array.\n\nEach object must have exactly these keys and types:\n  id      integer, 1 through 10, in order\n  name    non-empty string\n  active  boolean\n\nNo other keys. No nulls. Reply with the array only."
    }
  ],
  "max_tokens": 1700,
  "temperature": 0
}'