Code · v1
Locate an off-by-one error by line number
A short function with exactly one bug and two plausible decoys. The answer is a line number and nothing else, so the score measures whether the model found the defect rather than whether it can write convincingly about defects in general.
Results
Not yet run
No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.
This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.
Prompt
Exactly what every model receives. Nothing else is sent.
system
You are a code reviewer. When asked for a line number, reply with the number alone — no explanation, no sentence, no punctuation.
user
The function below returns the last `n` items of an array. It has exactly one bug.
Reply with the line number of the bug, as a number and nothing else.
1 function lastNItems(items, n) {
2 const out = [];
3 const start = items.length - n;
4
5 for (let i = start; i <= items.length; i++) {
6 out.push(items[i]);
7 }
8
9 return out;
10 }Rubric
Published so you can disagree with it. A score you cannot argue with is a rumour.
Exact match on "5". The loop condition reads one index past the end of the array, appending undefined. A negative `start` for n greater than items.length and the mutation of a const-declared array are both present, both harmless, and both wrong answers.
- Identifies line 5 — the loop bound, not the declaration on line 3
- Answers with the number alone
- No explanation, sentence or trailing punctuation
Scoring
Scored by exact match. Deterministic — the same output always produces the same score.
Expected output — surrounding whitespace ignored
5
Worked examples
Hand-written outputs the rubric is tested against on every build.
A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.
Correct — must score 1.00
5
Rejected — blamed the negative start — the plausible decoy
3
Rejected — blamed the push into a const array
6
Rejected — answered in a sentence
The bug is on line 5.
Rejected — labelled the answer
Line 5
Rejected — explained instead of locating
5 — the loop should use < rather than <=.
Version history
A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.
- v1 · current
- Initial version.
Parameters
- maxTokens
- 1020
- temperature
- 0
Run it yourself
The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You are a code reviewer. When asked for a line number, reply with the number alone — no explanation, no sentence, no punctuation."
},
{
"role": "user",
"content": "The function below returns the last `n` items of an array. It has exactly one bug.\n\nReply with the line number of the bug, as a number and nothing else.\n\n1 function lastNItems(items, n) {\n2 const out = [];\n3 const start = items.length - n;\n4\n5 for (let i = start; i <= items.length; i++) {\n6 out.push(items[i]);\n7 }\n8\n9 return out;\n10 }"
}
],
"max_tokens": 1020,
"temperature": 0
}'