Structured extraction · v1
Extract one contact from a signature containing several
An email signature holding two addresses, two phone numbers, a wrapped job title and a legal disclaimer. The schema wants one of each, so the task is choosing correctly rather than collecting everything. Tests the discrimination that decides whether a CRM record is usable or quietly points at a shared inbox.
Results
Not yet run
No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.
This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.
Prompt
Exactly what every model receives. Nothing else is sent.
system
You extract contact records. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences.
user
Extract the sender's own contact details from this email signature.
--
Dr Miriam Osei-Bonsu
Head of Platform
Engineering | Northwind Logistics
Direct: +44 20 7123 4567
Switchboard: +44 20 7100 0000
m.osei-bonsu@northwind-logistics.co.uk
Support: help@northwind-logistics.co.uk
Northwind Logistics Ltd, registered in England and Wales no. 08812234.
This email and any attachments are confidential and intended solely for the
addressee. If you have received it in error please notify the sender.
Schema:
{
full_name: string, including any title that appears
job_title: string, on one line
company: string, trading name only — no legal suffix
email: string, the sender's own address, not a shared alias
phone: string, E.164 format, the sender's direct line
}
Reply with the JSON only.Rubric
Published so you can disagree with it. A score you cannot argue with is a rumour.
Each field is compared against its exact expected value. The two traps are choosing the shared support alias over the sender's own address, and reporting the switchboard rather than the direct line. Both produce well-formed JSON that is wrong where it counts.
- Parses as JSON with no surrounding prose or code fence
- full_name keeps the title 'Dr'
- job_title rejoined across the line break, without the company name attached
- company is the trading name — 'Northwind Logistics', not 'Northwind Logistics Ltd'
- email is the sender's own address, not help@
- phone is the direct line in E.164, not the switchboard
Scoring
Scored by JSON Schema. Deterministic — the same output always produces the same score.
JSON Schema — the output must be bare JSON, no code fence
{
"type": "object",
"required": [
"full_name",
"job_title",
"company",
"email",
"phone"
],
"additionalProperties": false,
"properties": {
"full_name": {
"const": "Dr Miriam Osei-Bonsu"
},
"job_title": {
"const": "Head of Platform Engineering"
},
"company": {
"const": "Northwind Logistics"
},
"email": {
"const": "m.osei-bonsu@northwind-logistics.co.uk"
},
"phone": {
"const": "+442071234567"
}
}
}Worked examples
Hand-written outputs the rubric is tested against on every build.
A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.
Correct — must score 1.00
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform Engineering","company":"Northwind Logistics","email":"m.osei-bonsu@northwind-logistics.co.uk","phone":"+442071234567"}Rejected — chose the shared support alias — the subtle one
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform Engineering","company":"Northwind Logistics","email":"help@northwind-logistics.co.uk","phone":"+442071234567"}Rejected — took the switchboard instead of the direct line
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform Engineering","company":"Northwind Logistics","email":"m.osei-bonsu@northwind-logistics.co.uk","phone":"+442071000000"}Rejected — job title truncated at the line break
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform","company":"Northwind Logistics","email":"m.osei-bonsu@northwind-logistics.co.uk","phone":"+442071234567"}Rejected — phone left in its printed spacing rather than E.164
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform Engineering","company":"Northwind Logistics","email":"m.osei-bonsu@northwind-logistics.co.uk","phone":"+44 20 7123 4567"}Rejected — pulled the legal entity name out of the disclaimer
{"full_name":"Dr Miriam Osei-Bonsu","job_title":"Head of Platform Engineering","company":"Northwind Logistics Ltd","email":"m.osei-bonsu@northwind-logistics.co.uk","phone":"+442071234567"}Version history
A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.
- v1 · current
- Initial version.
Parameters
- maxTokens
- 1600
- temperature
- 0
Run it yourself
The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You extract contact records. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences."
},
{
"role": "user",
"content": "Extract the sender'\''s own contact details from this email signature.\n\n--\nDr Miriam Osei-Bonsu\nHead of Platform\nEngineering | Northwind Logistics\n\nDirect: +44 20 7123 4567\nSwitchboard: +44 20 7100 0000\nm.osei-bonsu@northwind-logistics.co.uk\nSupport: help@northwind-logistics.co.uk\n\nNorthwind Logistics Ltd, registered in England and Wales no. 08812234.\nThis email and any attachments are confidential and intended solely for the\naddressee. If you have received it in error please notify the sender.\n\nSchema:\n{\n full_name: string, including any title that appears\n job_title: string, on one line\n company: string, trading name only — no legal suffix\n email: string, the sender'\''s own address, not a shared alias\n phone: string, E.164 format, the sender'\''s direct line\n}\n\nReply with the JSON only."
}
],
"max_tokens": 1600,
"temperature": 0
}'