Structured extraction · v2
Extract invoice line items into strict JSON
Given a messy plain-text invoice, return a single JSON document matching a fixed schema. Tests whether a model can produce machine-consumable output without commentary, code fences, or invented fields — the failure mode that breaks a data pipeline at 3am.
Results
Not yet run
No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v2 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.
This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.
Prompt
Exactly what every model receives. Nothing else is sent.
system
You extract structured data. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences.
user
Extract this invoice into JSON matching the schema.
INVOICE INV-2291
Acme Supplies Ltd Date: 03/07/2026
--------------------------------------------
Widget, large qty two $ 24.50 ea
Bracket (steel) qty 10 $ 3.05 ea
Shipping $ 12.00
--------------------------------------------
TOTAL DUE $ 91.50
Note: payment terms net 30. Not a line item.
Schema: { invoice_number: string, currency: 3-letter code,
line_items: [{ description: string, quantity: number, unit_price: number }],
total: number }Rubric
Published so you can disagree with it. A score you cannot argue with is a rumour.
Output must be a bare JSON document satisfying the schema exactly. Code fences are not accepted. Scored deterministically — no judge model is involved.
- Parses as JSON with no surrounding prose or code fence
- Contains invoice_number, currency, line_items and total, and no other top-level keys
- currency is a 3-letter code, not a symbol
- Every line item has a numeric quantity — 'two' must become 2
- Shipping is handled without inventing fields outside the schema
Scoring
Scored by JSON Schema. Deterministic — the same output always produces the same score.
JSON Schema — the output must be bare JSON, no code fence
{
"type": "object",
"required": [
"invoice_number",
"currency",
"line_items",
"total"
],
"additionalProperties": false,
"properties": {
"invoice_number": {
"type": "string"
},
"currency": {
"type": "string",
"minLength": 3,
"maxLength": 3
},
"line_items": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": [
"description",
"quantity",
"unit_price"
],
"additionalProperties": false,
"properties": {
"description": {
"type": "string"
},
"quantity": {
"type": "number"
},
"unit_price": {
"type": "number"
}
}
}
},
"total": {
"type": "number"
}
}
}Worked examples
Hand-written outputs the rubric is tested against on every build.
A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.
Correct — must score 1.00
{"invoice_number":"INV-2291","currency":"USD","line_items":[{"description":"Widget, large","quantity":2,"unit_price":24.5},{"description":"Bracket (steel)","quantity":10,"unit_price":3.05},{"description":"Shipping","quantity":1,"unit_price":12}],"total":91.5}Rejected — prose instead of JSON
Here is the invoice data you asked for: the total is $91.50.
Rejected — wrapped in a code fence
```json
{"invoice_number":"INV-2291","currency":"USD","line_items":[{"description":"Widget","quantity":2,"unit_price":24.5}],"total":91.5}
```Rejected — quantity left as the word 'two' — the subtle one
{"invoice_number":"INV-2291","currency":"USD","line_items":[{"description":"Widget, large","quantity":"two","unit_price":24.5}],"total":91.5}Rejected — currency as a symbol rather than a code
{"invoice_number":"INV-2291","currency":"$","line_items":[{"description":"Widget","quantity":2,"unit_price":24.5}],"total":91.5}Rejected — invented an extra top-level field
{"invoice_number":"INV-2291","currency":"USD","line_items":[{"description":"Widget","quantity":2,"unit_price":24.5}],"total":91.5,"payment_terms":"net 30"}Version history
A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.
- v2 · current
- v2 — raised maxTokens by 1000 to leave room for model reasoning. Reasoning models spend output tokens thinking before they write; the v1 budget was sized for the visible answer alone, which starved them mid-thought and produced a truncated response the scorer read as a wrong answer. Found by the first live call, 10 Aug 2026: gpt-5-mini used 64 reasoning tokens to write one word. The prompt and rubric are unchanged — only the budget.
Parameters
- maxTokens
- 1800
- temperature
- 0
Run it yourself
The probe exactly as it stands at v2. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You extract structured data. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences."
},
{
"role": "user",
"content": "Extract this invoice into JSON matching the schema.\n\nINVOICE INV-2291\nAcme Supplies Ltd Date: 03/07/2026\n--------------------------------------------\nWidget, large qty two $ 24.50 ea\nBracket (steel) qty 10 $ 3.05 ea\nShipping $ 12.00\n--------------------------------------------\nTOTAL DUE $ 91.50\n\nNote: payment terms net 30. Not a line item.\n\nSchema: { invoice_number: string, currency: 3-letter code,\nline_items: [{ description: string, quantity: number, unit_price: number }],\ntotal: number }"
}
],
"max_tokens": 1800,
"temperature": 0
}'