Structured extraction · v1

Turn a raw log line into structured JSON

One raw proxy log line, converted to a fixed JSON shape. Tests the everyday ingestion job: reformatting a timestamp, lowercasing a level, and reading a status code that sits next to a byte count it is easy to confuse it with. Values are pinned exactly, because in a log pipeline a plausible wrong number is more damaging than a missing one.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You convert log lines into structured data. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences.

user

Convert this log line into JSON matching the schema.

[09/Aug/2026:14:02:57 +0000] ERROR checkout-api "POST /v1/orders HTTP/1.1" 504 30012ms 1180b upstream timed out

Schema:
{
  timestamp:  string, ISO 8601 UTC, e.g. "2026-01-31T09:05:00Z"
  level:      string, lowercase
  service:    string
  status:     number, the HTTP status code
  latency_ms: number, milliseconds, no unit suffix
  message:    string, the trailing free-text message only
}

Reply with the JSON only.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Every field is compared against its exact expected value, not merely its type. The timestamp must be converted to ISO 8601 UTC, the level lowercased, and `status` must be the HTTP status (504) rather than the latency (30012) or the byte count (1180).

Parses as JSON with no surrounding prose or code fence
timestamp converted from the bracketed Apache format to ISO 8601 UTC
level lowercased to 'error'
status is 504 — not the byte count, not the latency
latency_ms is the number 30012, with the 'ms' suffix stripped
message contains the trailing text only, not the request line

Scoring

Scored by JSON Schema. Deterministic — the same output always produces the same score.

JSON Schema — the output must be bare JSON, no code fence

{
  "type": "object",
  "required": [
    "timestamp",
    "level",
    "service",
    "status",
    "latency_ms",
    "message"
  ],
  "additionalProperties": false,
  "properties": {
    "timestamp": {
      "const": "2026-08-09T14:02:57Z"
    },
    "level": {
      "const": "error"
    },
    "service": {
      "const": "checkout-api"
    },
    "status": {
      "const": 504
    },
    "latency_ms": {
      "const": 30012
    },
    "message": {
      "const": "upstream timed out"
    }
  }
}

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

{"timestamp":"2026-08-09T14:02:57Z","level":"error","service":"checkout-api","status":504,"latency_ms":30012,"message":"upstream timed out"}

Rejected — left the timestamp in its original format

{"timestamp":"09/Aug/2026:14:02:57 +0000","level":"error","service":"checkout-api","status":504,"latency_ms":30012,"message":"upstream timed out"}

Rejected — read the byte count as the status — the subtle one

{"timestamp":"2026-08-09T14:02:57Z","level":"error","service":"checkout-api","status":1180,"latency_ms":30012,"message":"upstream timed out"}

Rejected — kept the unit suffix, making latency_ms a string

{"timestamp":"2026-08-09T14:02:57Z","level":"error","service":"checkout-api","status":504,"latency_ms":"30012ms","message":"upstream timed out"}

Rejected — left the level uppercase as it appeared in the line

{"timestamp":"2026-08-09T14:02:57Z","level":"ERROR","service":"checkout-api","status":504,"latency_ms":30012,"message":"upstream timed out"}

Rejected — swept the whole request line into message

{"timestamp":"2026-08-09T14:02:57Z","level":"error","service":"checkout-api","status":504,"latency_ms":30012,"message":"POST /v1/orders HTTP/1.1 504 upstream timed out"}

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v1 · current: Initial version.

Parameters

maxTokens: 1600
temperature: 0

Run it yourself

The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You convert log lines into structured data. Reply with a single JSON document and nothing else. No explanation, no markdown, no code fences."
    },
    {
      "role": "user",
      "content": "Convert this log line into JSON matching the schema.\n\n[09/Aug/2026:14:02:57 +0000] ERROR checkout-api \"POST /v1/orders HTTP/1.1\" 504 30012ms 1180b upstream timed out\n\nSchema:\n{\n  timestamp:  string, ISO 8601 UTC, e.g. \"2026-01-31T09:05:00Z\"\n  level:      string, lowercase\n  service:    string\n  status:     number, the HTTP status code\n  latency_ms: number, milliseconds, no unit suffix\n  message:    string, the trailing free-text message only\n}\n\nReply with the JSON only."
    }
  ],
  "max_tokens": 1600,
  "temperature": 0
}'