Structured extraction · v1

Return null for a field the document does not contain

A delivery note with no VAT number on it, and a company registration number printed two lines away that looks like one. The schema requires null. Tests whether a model will admit a field is absent instead of supplying the most plausible thing nearby — the failure that produces a structurally valid record asserting something untrue.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You extract structured data from documents. You report only what the document states. If a field is not present, its value is null. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences.

user

Extract this delivery note into JSON matching the schema.

HALLIWELL TIMBER
Delivery Note  DN-40218
Against order  PO-99471
Registered in England no. 07724119
-------------------------------------------------
 Oak board 18mm      12 sheets    @  84.20
 Pine batten 45x20   40 lengths   @   6.15
 Delivery                              78.00
-------------------------------------------------
 NET TOTAL                          1,842.60

Goods received in good condition. Signature on file.

Schema:
{
  supplier:              string, trading name
  delivery_note_number:  string
  order_reference:       string
  vat_number:            string, or null if the document does not state one
  net_total:             number
}

Reply with the JSON only.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

vat_number must be JSON null. A string of any kind fails, including "N/A", an empty string, and the company registration number 07724119 — which is present on the document but is not a VAT number. The other four fields are pinned to their exact values, so a model cannot pass by declining to extract anything.

Parses as JSON with no surrounding prose or code fence
vat_number is null, not a string
The registration number 07724119 is not reported as a VAT number
No placeholder text such as 'N/A', 'unknown' or '' in place of null
The four fields the document does state are extracted correctly
net_total is the number 1842.6, with the thousands separator removed

Scoring

Scored by JSON Schema. Deterministic — the same output always produces the same score.

JSON Schema — the output must be bare JSON, no code fence

{
  "type": "object",
  "required": [
    "supplier",
    "delivery_note_number",
    "order_reference",
    "vat_number",
    "net_total"
  ],
  "additionalProperties": false,
  "properties": {
    "supplier": {
      "const": "Halliwell Timber"
    },
    "delivery_note_number": {
      "const": "DN-40218"
    },
    "order_reference": {
      "const": "PO-99471"
    },
    "vat_number": {
      "type": "null"
    },
    "net_total": {
      "const": 1842.6
    }
  }
}

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":null,"net_total":1842.6}

Rejected — used the company registration number — the trap, and the most likely failure

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":"07724119","net_total":1842.6}

Rejected — invented a plausibly formatted VAT number

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":"GB077241190","net_total":1842.6}

Rejected — placeholder string instead of null

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":"N/A","net_total":1842.6}

Rejected — empty string instead of null

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":"","net_total":1842.6}

Rejected — omitted the key rather than reporting it as absent

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","net_total":1842.6}

Rejected — correct null, but stopped extracting anything else

{"supplier":"Halliwell Timber","delivery_note_number":"DN-40218","order_reference":"PO-99471","vat_number":null,"net_total":0}

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v1 · current: Initial version.

Parameters

maxTokens: 1600
temperature: 0

Run it yourself

The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You extract structured data from documents. You report only what the document states. If a field is not present, its value is null. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences."
    },
    {
      "role": "user",
      "content": "Extract this delivery note into JSON matching the schema.\n\nHALLIWELL TIMBER\nDelivery Note  DN-40218\nAgainst order  PO-99471\nRegistered in England no. 07724119\n-------------------------------------------------\n Oak board 18mm      12 sheets    @  84.20\n Pine batten 45x20   40 lengths   @   6.15\n Delivery                              78.00\n-------------------------------------------------\n NET TOTAL                          1,842.60\n\nGoods received in good condition. Signature on file.\n\nSchema:\n{\n  supplier:              string, trading name\n  delivery_note_number:  string\n  order_reference:       string\n  vat_number:            string, or null if the document does not state one\n  net_total:             number\n}\n\nReply with the JSON only."
    }
  ],
  "max_tokens": 1600,
  "temperature": 0
}'