LLM Ground

Long context · v1

Report a contradiction instead of answering from the first match

A long contract stating the same fee twice, with different numbers and no basis for choosing between them. Tests whether a model reads to the end before answering. Stopping at the first match produces a confident, well-sourced, wrong answer that a spot-check confirms.

Results

Not yet run

No model has been run against this probe yet, so there are no results to show. The definition, rubric and scoring below are complete and final for v1 — this page is published now so that the test is on the record before any score exists, rather than appearing alongside one.

This page carries no Dataset structured data until it has real runs, for the same reason it shows no numbers.

Prompt

Exactly what every model receives. Nothing else is sent.

system

You answer questions about contracts. You report what the document states, including where it is inconsistent. You do not resolve an inconsistency the document does not resolve.

user

Read the contract below and answer the question at the end.



Section 1. Routine operational notes for period 1. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 2. Routine operational notes for period 2. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 3. Routine operational notes for period 3. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 4. Routine operational notes for period 4. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 5. Routine operational notes for period 5. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 6. Routine operational notes for period 6. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 7. Routine operational notes for period 7. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 8. Routine operational notes for period 8. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 9. Routine operational notes for period 9. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 10. Routine operational notes for period 10. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 11. Routine operational notes for period 11. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 12. Commercial terms. The agreed early-termination fee is 22,500 EUR, payable within 30 days of written notice by either party.

Section 13. Routine operational notes for period 13. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 14. Routine operational notes for period 14. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 15. Routine operational notes for period 15. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 16. Routine operational notes for period 16. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 17. Routine operational notes for period 17. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 18. Routine operational notes for period 18. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 19. Routine operational notes for period 19. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 20. Routine operational notes for period 20. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 21. Routine operational notes for period 21. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 22. Routine operational notes for period 22. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 23. Routine operational notes for period 23. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 24. Routine operational notes for period 24. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 25. Routine operational notes for period 25. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 26. Routine operational notes for period 26. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 27. Routine operational notes for period 27. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 28. Routine operational notes for period 28. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 29. Routine operational notes for period 29. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 30. Routine operational notes for period 30. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 31. Routine operational notes for period 31. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 32. Routine operational notes for period 32. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 33. Routine operational notes for period 33. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 34. Routine operational notes for period 34. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 35. Routine operational notes for period 35. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 36. Routine operational notes for period 36. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 37. Routine operational notes for period 37. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 38. Routine operational notes for period 38. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 39. Routine operational notes for period 39. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 40. Routine operational notes for period 40. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 41. Schedule of charges. Early termination attracts a fee of 18,750 EUR, due on the final day of the notice period.

Section 42. Routine operational notes for period 42. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 43. Routine operational notes for period 43. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 44. Routine operational notes for period 44. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 45. Routine operational notes for period 45. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 46. Routine operational notes for period 46. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 47. Routine operational notes for period 47. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 48. Routine operational notes for period 48. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 49. Routine operational notes for period 49. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 50. Routine operational notes for period 50. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 51. Routine operational notes for period 51. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 52. Routine operational notes for period 52. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 53. Routine operational notes for period 53. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 54. Routine operational notes for period 54. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 55. Routine operational notes for period 55. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 56. Routine operational notes for period 56. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 57. Routine operational notes for period 57. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 58. Routine operational notes for period 58. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 59. Routine operational notes for period 59. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.

Section 60. Routine operational notes for period 60. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.



Question: what is the agreed early-termination fee?

Rubric

Published so you can disagree with it. A score you cannot argue with is a rumour.

Both stated figures must appear, and the response must name the disagreement rather than quote both in passing. No rule penalises expressing a view on which figure is likely to govern — the failure being measured is answering with one figure as though the other did not exist.

  • Reports the 22,500 EUR figure from section 12
  • Reports the 18,750 EUR figure from section 41
  • Explicitly identifies the two as conflicting, rather than listing both without comment

Scoring

Scored by regex rules. Deterministic — the same output always produces the same score.

Rules, in order

reports_first_figure
  must match  /22[,.]?500/
  Section 12's figure. Missing it means the document was not read to the end, or from the start.

reports_second_figure
  must match  /18[,.]?750/
  Section 41's figure. This is the one a model misses by answering from the first match, which is the specific failure this probe exists to catch.

names_the_conflict
  must match  /(conflict|contradict|inconsisten|discrepan|disagree|do(es)? not match|two different|differing|differs|ambigu)/i
  Quoting both figures without saying they disagree leaves the reader to notice the problem themselves, which is the work the question asked for.

Worked examples

Hand-written outputs the rubric is tested against on every build.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

Correct — must score 1.00

The contract states the early-termination fee twice, and the two figures conflict.

Section 12 (Commercial terms) gives 22,500 EUR, payable within 30 days of written notice.
Section 41 (Schedule of charges) gives 18,750 EUR, due on the final day of the notice period.

Nothing in the document indicates which governs — neither section is dated and neither
refers to the other. This needs clarifying with the counterparty before the figure is
relied on.

Rejected — answered from the first match — confident, sourced, and wrong

The agreed early-termination fee is 22,500 EUR, payable within 30 days of written notice by either party (section 12).

Rejected — found the later figure only

Early termination attracts a fee of 18,750 EUR, due on the final day of the notice period.

Rejected — quoted both without noticing they disagree

The contract mentions early-termination charges in two places. Section 12 refers to 22,500 EUR and section 41 refers to 18,750 EUR. Both relate to ending the agreement early.

Rejected — averaged them into a figure the document never states

The early-termination fee is approximately 20,625 EUR based on the terms given.

Version history

A probe version is immutable. Changing a prompt or a rubric creates the next version; existing runs stay attached to the one that produced them.

v1 · current
Initial version.

Parameters

maxTokens
1500
temperature
0

Run it yourself

The probe exactly as it stands at v1. Swap the model slug for any model you want to compare — the API key is a shell variable, never a value.

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You answer questions about contracts. You report what the document states, including where it is inconsistent. You do not resolve an inconsistency the document does not resolve."
    },
    {
      "role": "user",
      "content": "Read the contract below and answer the question at the end.\n\n\n\nSection 1. Routine operational notes for period 1. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 2. Routine operational notes for period 2. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 3. Routine operational notes for period 3. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 4. Routine operational notes for period 4. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 5. Routine operational notes for period 5. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 6. Routine operational notes for period 6. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 7. Routine operational notes for period 7. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 8. Routine operational notes for period 8. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 9. Routine operational notes for period 9. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 10. Routine operational notes for period 10. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 11. Routine operational notes for period 11. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 12. Commercial terms. The agreed early-termination fee is 22,500 EUR, payable within 30 days of written notice by either party.\n\nSection 13. Routine operational notes for period 13. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 14. Routine operational notes for period 14. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 15. Routine operational notes for period 15. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 16. Routine operational notes for period 16. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 17. Routine operational notes for period 17. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 18. Routine operational notes for period 18. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 19. Routine operational notes for period 19. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 20. Routine operational notes for period 20. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 21. Routine operational notes for period 21. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 22. Routine operational notes for period 22. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 23. Routine operational notes for period 23. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 24. Routine operational notes for period 24. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 25. Routine operational notes for period 25. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 26. Routine operational notes for period 26. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 27. Routine operational notes for period 27. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 28. Routine operational notes for period 28. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 29. Routine operational notes for period 29. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 30. Routine operational notes for period 30. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 31. Routine operational notes for period 31. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 32. Routine operational notes for period 32. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 33. Routine operational notes for period 33. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 34. Routine operational notes for period 34. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 35. Routine operational notes for period 35. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 36. Routine operational notes for period 36. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 37. Routine operational notes for period 37. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 38. Routine operational notes for period 38. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 39. Routine operational notes for period 39. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 40. Routine operational notes for period 40. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 41. Schedule of charges. Early termination attracts a fee of 18,750 EUR, due on the final day of the notice period.\n\nSection 42. Routine operational notes for period 42. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 43. Routine operational notes for period 43. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 44. Routine operational notes for period 44. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 45. Routine operational notes for period 45. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 46. Routine operational notes for period 46. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 47. Routine operational notes for period 47. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 48. Routine operational notes for period 48. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 49. Routine operational notes for period 49. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 50. Routine operational notes for period 50. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 51. Routine operational notes for period 51. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 52. Routine operational notes for period 52. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 53. Routine operational notes for period 53. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 54. Routine operational notes for period 54. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 55. Routine operational notes for period 55. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 56. Routine operational notes for period 56. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 57. Routine operational notes for period 57. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 58. Routine operational notes for period 58. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 59. Routine operational notes for period 59. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\nSection 60. Routine operational notes for period 60. Throughput remained within expected bounds and no incidents were raised. Staffing levels unchanged. Vendor review deferred to the following quarter.\n\n\n\nQuestion: what is the agreed early-termination fee?"
    }
  ],
  "max_tokens": 1500,
  "temperature": 0
}'