Long context · v1
Obey a format instruction given after a long document
The fact is in section 3; the output contract is in the final line, fifty sections later. Tests whether an instruction placed after a long input still governs the answer — a different capability from finding the fact, and the one that decides whether a long prompt can be trusted to end with its own rules.
Results
Not yet run
No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.
This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.
Prompt
Exactly what every model receives. Nothing else is sent.
system
You answer questions about internal documents clearly and helpfully.
user
Read the handover document below in full, then follow the instruction at the end. Section 1. Handover notes. Routine items for area 1 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 2. Handover notes. Routine items for area 2 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 3. Handover notes. Migration ownership transferred to Priya Raghunathan on 4 March, effective immediately. All schema change requests route through her from that date, and the previous owner retains read access only. Section 4. Handover notes. Routine items for area 4 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 5. Handover notes. Routine items for area 5 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 6. Handover notes. Routine items for area 6 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 7. Handover notes. Routine items for area 7 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 8. Handover notes. Routine items for area 8 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 9. Handover notes. Routine items for area 9 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 10. Handover notes. Routine items for area 10 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 11. Handover notes. Routine items for area 11 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 12. Handover notes. Routine items for area 12 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 13. Handover notes. Routine items for area 13 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 14. Handover notes. Routine items for area 14 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 15. Handover notes. Routine items for area 15 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 16. Handover notes. Routine items for area 16 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 17. Handover notes. Routine items for area 17 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 18. Handover notes. Routine items for area 18 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 19. Handover notes. Routine items for area 19 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 20. Handover notes. Routine items for area 20 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 21. Handover notes. Routine items for area 21 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 22. Handover notes. Routine items for area 22 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 23. Handover notes. Routine items for area 23 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 24. Handover notes. Routine items for area 24 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 25. Handover notes. Routine items for area 25 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 26. Handover notes. Routine items for area 26 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 27. Handover notes. Routine items for area 27 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 28. Handover notes. Routine items for area 28 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 29. Handover notes. Routine items for area 29 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 30. Handover notes. Routine items for area 30 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 31. Handover notes. Routine items for area 31 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 32. Handover notes. Routine items for area 32 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 33. Handover notes. Routine items for area 33 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 34. Handover notes. Routine items for area 34 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 35. Handover notes. Routine items for area 35 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 36. Handover notes. Routine items for area 36 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 37. Handover notes. Routine items for area 37 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 38. Handover notes. Routine items for area 38 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 39. Handover notes. Routine items for area 39 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 40. Handover notes. Routine items for area 40 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 41. Handover notes. Routine items for area 41 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 42. Handover notes. Routine items for area 42 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 43. Handover notes. Routine items for area 43 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 44. Handover notes. Routine items for area 44 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 45. Handover notes. Routine items for area 45 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 46. Handover notes. Routine items for area 46 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 47. Handover notes. Routine items for area 47 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 48. Handover notes. Routine items for area 48 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 49. Handover notes. Routine items for area 49 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Section 50. Handover notes. Routine items for area 50 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected. Question: who owns the migration? Reply with that person's surname only, in lowercase, with no other text, no punctuation and no sentence.
Rubric
Published so you can disagree with it. A score you cannot argue with is a rumour.
Exact match on "raghunathan", case-insensitively, after trimming. Retrieving the right person and answering in a sentence fails, which is the point: this measures whether the closing instruction survived the document, not whether the fact was found.
- Retrieves the name from section 3, fifty sections above the question
- Returns the surname only — not the full name
- Lowercase, as the closing line requires
- No sentence, label or trailing punctuation
Scoring
Scored by exact match. Deterministic — the same output always produces the same score.
Expected output — surrounding whitespace ignored, case-insensitive
raghunathan
Worked examples
Hand-written outputs the rubric is tested against on every build.
A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.
Correct — must score 1.00
raghunathan
Rejected — found the fact, ignored the closing instruction
The migration is owned by Priya Raghunathan.
Rejected — full name instead of surname
Priya Raghunathan
Rejected — gave the first name
priya
Rejected — surname with trailing punctuation
raghunathan.
Rejected — labelled the answer
Owner: raghunathan
Rejected — could not locate it across the span
The document does not say who owns the migration.
Version history
A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.
- v1 · current
- Initial version.
Parameters
- maxTokens
- 1040
- temperature
- 0
Run it yourself
The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.
curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You answer questions about internal documents clearly and helpfully."
},
{
"role": "user",
"content": "Read the handover document below in full, then follow the instruction at the end.\n\n\n\nSection 1. Handover notes. Routine items for area 1 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 2. Handover notes. Routine items for area 2 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 3. Handover notes. Migration ownership transferred to Priya Raghunathan on 4 March, effective immediately. All schema change requests route through her from that date, and the previous owner retains read access only.\n\nSection 4. Handover notes. Routine items for area 4 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 5. Handover notes. Routine items for area 5 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 6. Handover notes. Routine items for area 6 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 7. Handover notes. Routine items for area 7 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 8. Handover notes. Routine items for area 8 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 9. Handover notes. Routine items for area 9 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 10. Handover notes. Routine items for area 10 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 11. Handover notes. Routine items for area 11 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 12. Handover notes. Routine items for area 12 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 13. Handover notes. Routine items for area 13 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 14. Handover notes. Routine items for area 14 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 15. Handover notes. Routine items for area 15 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 16. Handover notes. Routine items for area 16 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 17. Handover notes. Routine items for area 17 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 18. Handover notes. Routine items for area 18 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 19. Handover notes. Routine items for area 19 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 20. Handover notes. Routine items for area 20 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 21. Handover notes. Routine items for area 21 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 22. Handover notes. Routine items for area 22 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 23. Handover notes. Routine items for area 23 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 24. Handover notes. Routine items for area 24 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 25. Handover notes. Routine items for area 25 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 26. Handover notes. Routine items for area 26 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 27. Handover notes. Routine items for area 27 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 28. Handover notes. Routine items for area 28 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 29. Handover notes. Routine items for area 29 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 30. Handover notes. Routine items for area 30 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 31. Handover notes. Routine items for area 31 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 32. Handover notes. Routine items for area 32 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 33. Handover notes. Routine items for area 33 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 34. Handover notes. Routine items for area 34 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 35. Handover notes. Routine items for area 35 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 36. Handover notes. Routine items for area 36 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 37. Handover notes. Routine items for area 37 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 38. Handover notes. Routine items for area 38 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 39. Handover notes. Routine items for area 39 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 40. Handover notes. Routine items for area 40 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 41. Handover notes. Routine items for area 41 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 42. Handover notes. Routine items for area 42 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 43. Handover notes. Routine items for area 43 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 44. Handover notes. Routine items for area 44 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 45. Handover notes. Routine items for area 45 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 46. Handover notes. Routine items for area 46 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 47. Handover notes. Routine items for area 47 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 48. Handover notes. Routine items for area 48 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 49. Handover notes. Routine items for area 49 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\nSection 50. Handover notes. Routine items for area 50 were reviewed and nothing was outstanding. Access lists unchanged. Documentation links verified. Rota unaffected.\n\n\n\nQuestion: who owns the migration?\n\n\n\nReply with that person'\''s surname only, in lowercase, with no other text, no punctuation\n\nand no sentence."
}
],
"max_tokens": 1040,
"temperature": 0
}'