Instruction following · v3 · Last run 2026-08-10

Answer with exactly one word, and nothing else

A trivial classification question with a strict output contract: one lowercase word, no punctuation, no preamble. The knowledge required is nil, so the score measures only whether the model can stop talking — the property that decides whether its output can be parsed instead of read.

Results

Every published run of this probe. Each row links to its raw output.

Probe v2 · superseded

1 run · mean 0.00

These runs used an earlier definition of this probe. They are kept separate rather than merged into the table above, because models measured against a different prompt were not measured against the same test.

Model	Score	Cost	Latency	Run on	Evidence
Nemotron 3 Nano 30B (free)nvidia/nemotron-3-nano-30b-a3b:free	0.00failed	$0	392 ms	2026-08-10	run #19 →

Prompt

Exactly what every model receives. Nothing else is sent.

system

You are a classifier. Reply with exactly one lowercase word. No punctuation, no explanation, no preamble.

user

Classify the sentiment of this review as exactly one of: positive, negative, neutral.

Review: "Arrived broken, nobody replied to three support emails, and I still have not had the refund I was promised."

Reply with one word.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Exact match against "negative" after trimming surrounding whitespace. Any added punctuation, capitalisation or commentary fails. The review is deliberately unambiguous — broken goods, no support response, no refund — so the classification carries no judgement and the score reflects the output contract alone.

Output is exactly the word 'negative'
No trailing period, quotes or markdown
No preamble such as 'Sentiment:' or 'Sure!'

Scoring

Scored by exact match. Deterministic — the same output always produces the same score.

Expected output — surrounding whitespace ignored, case-insensitive

negative

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

negative

Rejected — trailing period

negative.

Rejected — labelled the answer

Sentiment: negative

Rejected — explained itself

negative — broken goods, no support reply and no refund.

Rejected — conversational preamble

Sure! The sentiment is negative.

Rejected — wrapped in quotes

"negative"

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v3 · current: v3 — replaced an ambiguous review with an unambiguous one. v2 asked models to classify "Arrived two days late and the box was crushed, but the item itself works fine" and expected "negative". That review is genuinely mixed, so "neutral" was defensible — and the first published run (run 19, nvidia/nemotron-3-nano-30b-a3b, 10 Aug 2026) returned exactly that: one lowercase word, seven bytes, no punctuation, no preamble, scored 0. The format contract was met perfectly and the probe failed it on an arguable judgement it never claimed to be testing. The new review contains nothing redeeming, so the only way to score 0 is to break the output contract. Scoring, rubric and params are otherwise unchanged; v2 runs remain attached to v2 and still mean what they meant.

Parameters

maxTokens: 1020
temperature: 0

Run it yourself

The probe exactly as it stands at v3. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You are a classifier. Reply with exactly one lowercase word. No punctuation, no explanation, no preamble."
    },
    {
      "role": "user",
      "content": "Classify the sentiment of this review as exactly one of: positive, negative, neutral.\n\nReview: \"Arrived broken, nobody replied to three support emails, and I still have not had the refund I was promised.\"\n\nReply with one word."
    }
  ],
  "max_tokens": 1020,
  "temperature": 0
}'