Structured extraction · v2

Convert CSV to JSON without stringifying numbers

Convert four CSV rows into JSON with correct primitive types. Easy to read, easy to get subtly wrong: a quantity emitted as "12" instead of 12 parses fine, passes a loose schema, and breaks arithmetic somewhere else entirely.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v2 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You convert tabular data to JSON. Reply with a single JSON array and nothing else. Preserve correct primitive types: numbers as numbers, booleans as booleans.

user

Convert to a JSON array of objects with keys: sku, qty, price, in_stock.

sku,qty,price,in_stock
0042-A,12,$4.99,yes
0100-B,3,$1,299.00,no
X-7,0,$0.50,yes
0007-C,1000,$12.00,no

Note: SKUs are identifiers, not numbers.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

A bare JSON array of exactly four objects. qty must be an integer, price a number, in_stock a boolean, sku a string with its leading zeros intact.

Parses as JSON with no prose or code fence
Exactly four objects, one per data row
qty is an integer, not a quoted string
price is a number with the currency symbol and thousands separator removed
in_stock is a real boolean, not the string "yes"
sku stays a string — 0042-A must not become 42

Scoring

Scored by JSON Schema. Deterministic — the same output always produces the same score.

JSON Schema — the output must be bare JSON, no code fence

{
  "type": "array",
  "minItems": 4,
  "maxItems": 4,
  "items": {
    "type": "object",
    "required": [
      "sku",
      "qty",
      "price",
      "in_stock"
    ],
    "additionalProperties": false,
    "properties": {
      "sku": {
        "type": "string"
      },
      "qty": {
        "type": "integer"
      },
      "price": {
        "type": "number"
      },
      "in_stock": {
        "type": "boolean"
      }
    }
  }
}

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

[{"sku":"0042-A","qty":12,"price":4.99,"in_stock":true},{"sku":"0100-B","qty":3,"price":1299,"in_stock":false},{"sku":"X-7","qty":0,"price":0.5,"in_stock":true},{"sku":"0007-C","qty":1000,"price":12,"in_stock":false}]

Rejected — numbers stringified — the failure this probe exists for

[{"sku":"0042-A","qty":"12","price":"4.99","in_stock":true},{"sku":"0100-B","qty":"3","price":"1299.00","in_stock":false},{"sku":"X-7","qty":"0","price":"0.50","in_stock":true},{"sku":"0007-C","qty":"1000","price":"12.00","in_stock":false}]

Rejected — booleans left as yes/no strings

[{"sku":"0042-A","qty":12,"price":4.99,"in_stock":"yes"},{"sku":"0100-B","qty":3,"price":1299,"in_stock":"no"},{"sku":"X-7","qty":0,"price":0.5,"in_stock":"yes"},{"sku":"0007-C","qty":1000,"price":12,"in_stock":"no"}]

Rejected — dropped a row

[{"sku":"0042-A","qty":12,"price":4.99,"in_stock":true},{"sku":"0100-B","qty":3,"price":1299,"in_stock":false},{"sku":"X-7","qty":0,"price":0.5,"in_stock":true}]

Rejected — currency symbol left in the price

[{"sku":"0042-A","qty":12,"price":"$4.99","in_stock":true},{"sku":"0100-B","qty":3,"price":"$1299.00","in_stock":false},{"sku":"X-7","qty":0,"price":"$0.50","in_stock":true},{"sku":"0007-C","qty":1000,"price":"$12.00","in_stock":false}]

Rejected — explanatory sentence before the array

Here is the converted data:
[{"sku":"0042-A","qty":12,"price":4.99,"in_stock":true}]

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v2 · current: v2 — raised maxTokens by 1000 to leave room for model reasoning. Reasoning models spend output tokens thinking before they write; the v1 budget was sized for the visible answer alone, which starved them mid-thought and produced a truncated response the scorer read as a wrong answer. Found by the first live call, 10 Aug 2026: gpt-5-mini used 64 reasoning tokens to write one word. The prompt and rubric are unchanged — only the budget.

Parameters

maxTokens: 1600
temperature: 0

Run it yourself

The probe exactly as it stands at v2. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You convert tabular data to JSON. Reply with a single JSON array and nothing else. Preserve correct primitive types: numbers as numbers, booleans as booleans."
    },
    {
      "role": "user",
      "content": "Convert to a JSON array of objects with keys: sku, qty, price, in_stock.\n\nsku,qty,price,in_stock\n0042-A,12,$4.99,yes\n0100-B,3,$1,299.00,no\nX-7,0,$0.50,yes\n0007-C,1000,$12.00,no\n\nNote: SKUs are identifiers, not numbers."
    }
  ],
  "max_tokens": 1600,
  "temperature": 0
}'