Long context · v1
Count every occurrence of a marker across a long document
Forty duty-log sections, seven of them marked ESCALATED. Counting requires holding every occurrence at once rather than finding one, and the characteristic failure is a near-miss — six or eight — which is more dangerous than a wild guess because a plausible count gets used.
Results
Not yet run
No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.
This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.
Prompt
Exactly what every model receives. Nothing else is sent.
system
You answer questions about documents. When asked for a number, reply with digits only — no sentence, no working, no units.
user
Read the duty log below and answer the question at the end. Section 1. Duty log for period 1. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 2. Duty log for period 2. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 3. Duty log for period 3. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 4. Duty log for period 4. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 5. Duty log for period 5. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 6. Duty log for period 6. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 7. Duty log for period 7. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 8. Duty log for period 8. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 9. Duty log for period 9. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 10. Duty log for period 10. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 11. Duty log for period 11. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 12. Duty log for period 12. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 13. Duty log for period 13. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 14. Duty log for period 14. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 15. Duty log for period 15. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 16. Duty log for period 16. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 17. Duty log for period 17. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 18. Duty log for period 18. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 19. Duty log for period 19. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 20. Duty log for period 20. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 21. Duty log for period 21. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 22. Duty log for period 22. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 23. Duty log for period 23. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 24. Duty log for period 24. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 25. Duty log for period 25. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 26. Duty log for period 26. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 27. Duty log for period 27. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 28. Duty log for period 28. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 29. Duty log for period 29. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 30. Duty log for period 30. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 31. Duty log for period 31. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 32. Duty log for period 32. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 33. Duty log for period 33. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 34. Duty log for period 34. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 35. Duty log for period 35. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 36. Duty log for period 36. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 37. Duty log for period 37. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 38. Duty log for period 38. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Section 39. Duty log for period 39. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED. Section 40. Duty log for period 40. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED. Question: how many sections have the status ESCALATED? Reply with the number in digits and nothing else.
Rubric
Published so you can disagree with it. A score you cannot argue with is a rumour.
Exact match on "7". The marker ESCALATED appears in exactly seven of the forty sections and nowhere else in the document; the filler avoids the word in every form, so the count is not open to interpretation.
- Counts all seven occurrences across the full document
- Does not stop counting partway through
- Answers with digits alone, with no sentence or working shown
Scoring
Scored by exact match. Deterministic — the same output always produces the same score.
Expected output — surrounding whitespace ignored
7
Worked examples
Hand-written outputs the rubric is tested against on every build.
A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.
Correct — must score 1.00
7
Rejected — near miss — the characteristic counting failure
6
Rejected — near miss in the other direction
8
Rejected — correct count, wrong format
There are 7 sections marked ESCALATED.
Rejected — spelled the number
seven
Rejected — showed its working
Sections 4, 9, 17, 23, 28, 35 and 39 — that is 7.
Version history
A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.
- v1 · current
- Initial version.
Parameters
- maxTokens
- 1040
- temperature
- 0
Run it yourself
The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You answer questions about documents. When asked for a number, reply with digits only — no sentence, no working, no units."
},
{
"role": "user",
"content": "Read the duty log below and answer the question at the end.\n\n\n\nSection 1. Duty log for period 1. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 2. Duty log for period 2. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 3. Duty log for period 3. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 4. Duty log for period 4. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 5. Duty log for period 5. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 6. Duty log for period 6. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 7. Duty log for period 7. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 8. Duty log for period 8. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 9. Duty log for period 9. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 10. Duty log for period 10. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 11. Duty log for period 11. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 12. Duty log for period 12. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 13. Duty log for period 13. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 14. Duty log for period 14. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 15. Duty log for period 15. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 16. Duty log for period 16. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 17. Duty log for period 17. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 18. Duty log for period 18. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 19. Duty log for period 19. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 20. Duty log for period 20. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 21. Duty log for period 21. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 22. Duty log for period 22. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 23. Duty log for period 23. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 24. Duty log for period 24. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 25. Duty log for period 25. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 26. Duty log for period 26. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 27. Duty log for period 27. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 28. Duty log for period 28. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 29. Duty log for period 29. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 30. Duty log for period 30. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 31. Duty log for period 31. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 32. Duty log for period 32. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 33. Duty log for period 33. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 34. Duty log for period 34. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 35. Duty log for period 35. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 36. Duty log for period 36. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 37. Duty log for period 37. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 38. Duty log for period 38. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\nSection 39. Duty log for period 39. Response time exceeded the agreed threshold and the matter was passed to the on-call lead. Status: ESCALATED.\n\nSection 40. Duty log for period 40. All checks completed within the agreed threshold and the shift closed without follow-up. Status: CLOSED.\n\n\n\nQuestion: how many sections have the status ESCALATED?\n\n\n\nReply with the number in digits and nothing else."
}
],
"max_tokens": 1040,
"temperature": 0
}'