Long context · v2

Retrieve one buried fact from a long document

Sixty near-identical sections of routine notes with a single contract figure buried three-quarters of the way through. Tests retrieval across a long span — the deliberate long-context exception to this library's keep-inputs-short rule.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v2 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You answer questions about documents. Reply with only the value asked for — no explanation, no units beyond what is asked, no sentence.

user

Read the document below and answer the question at the end.

Section 1. Routine operational notes for period 1. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 2. Routine operational notes for period 2. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 3. Routine operational notes for period 3. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 4. Routine operational notes for period 4. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 5. Routine operational notes for period 5. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 6. Routine operational notes for period 6. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 7. Routine operational notes for period 7. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 8. Routine operational notes for period 8. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 9. Routine operational notes for period 9. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 10. Routine operational notes for period 10. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 11. Routine operational notes for period 11. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 12. Routine operational notes for period 12. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 13. Routine operational notes for period 13. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 14. Routine operational notes for period 14. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 15. Routine operational notes for period 15. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 16. Routine operational notes for period 16. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 17. Routine operational notes for period 17. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 18. Routine operational notes for period 18. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 19. Routine operational notes for period 19. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 20. Routine operational notes for period 20. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 21. Routine operational notes for period 21. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 22. Routine operational notes for period 22. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 23. Routine operational notes for period 23. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 24. Routine operational notes for period 24. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 25. Routine operational notes for period 25. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 26. Routine operational notes for period 26. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 27. Routine operational notes for period 27. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 28. Routine operational notes for period 28. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 29. Routine operational notes for period 29. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 30. Routine operational notes for period 30. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 31. Routine operational notes for period 31. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 32. Routine operational notes for period 32. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 33. Routine operational notes for period 33. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 34. Routine operational notes for period 34. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 35. Routine operational notes for period 35. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 36. Routine operational notes for period 36. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 37. Routine operational notes for period 37. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 38. Routine operational notes for period 38. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 39. Routine operational notes for period 39. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 40. Routine operational notes for period 40. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 41. Routine operational notes for period 41. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 42. Routine operational notes for period 42. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 43. Routine operational notes for period 43. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 44. Routine operational notes for period 44. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 45. Contract addendum: the agreed early-termination fee is 17,400 GBP, payable within 30 days of notice.

Section 46. Routine operational notes for period 46. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 47. Routine operational notes for period 47. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 48. Routine operational notes for period 48. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 49. Routine operational notes for period 49. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 50. Routine operational notes for period 50. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 51. Routine operational notes for period 51. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 52. Routine operational notes for period 52. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 53. Routine operational notes for period 53. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 54. Routine operational notes for period 54. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 55. Routine operational notes for period 55. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 56. Routine operational notes for period 56. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 57. Routine operational notes for period 57. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 58. Routine operational notes for period 58. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 59. Routine operational notes for period 59. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 60. Routine operational notes for period 60. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.

Question: What is the agreed early-termination fee? Reply with only the number and currency code, formatted exactly as it appears in the document.

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Exact match on "17,400 GBP" after trimming. The document states the figure in that form and the prompt asks for it exactly as it appears, so reformatting fails.

Locates the figure despite sixty near-identical surrounding sections
Reproduces it exactly as written, including the thousands separator
Adds no sentence, unit gloss or explanation

Scoring

Scored by exact match. Deterministic — the same output always produces the same score.

Expected output — surrounding whitespace ignored, case-insensitive

17,400 GBP

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

17,400 GBP

Rejected — reformatted the number

17400 GBP

Rejected — currency symbol instead of code

£17,400

Rejected — answered in a sentence

The agreed early-termination fee is 17,400 GBP.

Rejected — wrong figure — plausible hallucination

14,700 GBP

Rejected — could not find it

The document does not state an early-termination fee.

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v2 · current: v2 — raised maxTokens by 1000 to leave room for model reasoning. Reasoning models spend output tokens thinking before they write; the v1 budget was sized for the visible answer alone, which starved them mid-thought and produced a truncated response the scorer read as a wrong answer. Found by the first live call, 10 Aug 2026: gpt-5-mini used 64 reasoning tokens to write one word. The prompt and rubric are unchanged — only the budget.

Parameters

maxTokens: 1040
temperature: 0

Run it yourself

The probe exactly as it stands at v2. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-opus-5",
"messages": [
{
"role": "system",
"content": "You answer questions about documents. Reply with only the value asked for — no explanation, no units beyond what is asked, no sentence."
},
{
"role": "user",
"content": "Read the document below and answer the question at the end.\n\nSection 1. Routine operational notes for period 1. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 2. Routine operational notes for period 2. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 3. Routine operational notes for period 3. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 4. Routine operational notes for period 4. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 5. Routine operational notes for period 5. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 6. Routine operational notes for period 6. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 7. Routine operational notes for period 7. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 8. Routine operational notes for period 8. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 9. Routine operational notes for period 9. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 10. Routine operational notes for period 10. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 11. Routine operational notes for period 11. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 12. Routine operational notes for period 12. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 13. Routine operational notes for period 13. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 14. Routine operational notes for period 14. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 15. Routine operational notes for period 15. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 16. Routine operational notes for period 16. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 17. Routine operational notes for period 17. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 18. Routine operational notes for period 18. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 19. Routine operational notes for period 19. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 20. Routine operational notes for period 20. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 21. Routine operational notes for period 21. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 22. Routine operational notes for period 22. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 23. Routine operational notes for period 23. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 24. Routine operational notes for period 24. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 25. Routine operational notes for period 25. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 26. Routine operational notes for period 26. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 27. Routine operational notes for period 27. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 28. Routine operational notes for period 28. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 29. Routine operational notes for period 29. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 30. Routine operational notes for period 30. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 31. Routine operational notes for period 31. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 32. Routine operational notes for period 32. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 33. Routine operational notes for period 33. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 34. Routine operational notes for period 34. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 35. Routine operational notes for period 35. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 36. Routine operational notes for period 36. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 37. Routine operational notes for period 37. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 38. Routine operational notes for period 38. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 39. Routine operational notes for period 39. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 40. Routine operational notes for period 40. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 41. Routine operational notes for period 41. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 42. Routine operational notes for period 42. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 43. Routine operational notes for period 43. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 44. Routine operational notes for period 44. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 45. Contract addendum: the agreed early-termination fee is 17,400 GBP, payable within 30 days of notice.\n\nSection 46. Routine operational notes for period 46. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 47. Routine operational notes for period 47. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 48. Routine operational notes for period 48. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 49. Routine operational notes for period 49. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 50. Routine operational notes for period 50. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 51. Routine operational notes for period 51. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 52. Routine operational notes for period 52. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 53. Routine operational notes for period 53. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 54. Routine operational notes for period 54. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 55. Routine operational notes for period 55. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 56. Routine operational notes for period 56. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 57. Routine operational notes for period 57. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 58. Routine operational notes for period 58. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 59. Routine operational notes for period 59. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 60. Routine operational notes for period 60. Throughput remained within expected bounds and no incidents were escalated. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nQuestion: What is the agreed early-termination fee? Reply with only the number and currency code, formatted exactly as it appears in the document."
}
],
"max_tokens": 1040,
"temperature": 0
}'