Diagnose a stack trace without blaming the wrong thing

system

You are a backend engineer debugging a production error. You reason from the evidence you are given and you do not assert causes the evidence does not support.

user

Diagnose this production error in under 120 words. State the root cause and the fix.

ERROR
TypeError: Cannot read properties of undefined (reading 'customer')
    at buildInvoice (/srv/app/invoice.js:14:31)
    at processBatch (/srv/app/batch.js:52:20)
    at async main (/srv/app/index.js:9:3)

CODE — /srv/app/invoice.js
10  async function buildInvoice(accountId) {
11    const orders = await db.orders.findMany({
12      where: { accountId, status: 'PENDING' }
13    });
14    const customerName = orders[0].customer.name;
15    return renderInvoice(customerName, orders);
16  }

WHAT WE KNOW
- The account being processed when this threw has no pending orders.
- db.orders.findMany returns an array, and returns [] when nothing matches.
- The same code has run successfully for every other account in the batch.
- No database errors were logged during the incident.

The root cause is that findMany returned an empty array, so orders[0] is undefined and reading .customer on it throws. The error message names 'customer', which makes 'the order is missing a customer field' the tempting and incorrect answer. A fix must guard the empty-result case; adding optional chaining to .customer alone silences the symptom and leaves customerName undefined.

• identifies_empty_array_as_the_cause
• does_not_blame_a_missing_customer_field
• does_not_invent_an_unstated_cause
• fix_addresses_the_empty_result_rather_than_the_symptom
• stays_within_length

This probe uses judge scoring with no additional configuration.

You are grading a production-error diagnosis written by another model.

The correct root cause: db.orders.findMany returned an empty array for an account with no
pending orders, so orders[0] evaluates to undefined, and reading .customer on undefined
throws. The wording will vary — accept any phrasing that identifies the empty result or the
undefined array element as the cause.

The attractive wrong answer is 'the order object is missing a customer field'. The message
names 'customer', which makes this read plausibly, but the trace says the error is reading
'customer' OF undefined, so the undefined value is orders[0], not orders[0].customer. Mark
does_not_blame_a_missing_customer_field as a FAILURE if the diagnosis rests on this.

Treat any cause not supported by the supplied facts as invented — a database outage, a
race condition, a schema migration, a null customer row. No database errors were logged and
the same code succeeded for every other account.

For the fix criterion: a guard on the empty result (checking length, using a default, or
handling the no-orders case explicitly) passes. Optional chaining applied only to .customer
fails — it stops the throw and leaves customerName undefined, moving the failure downstream.

Grade the reasoning, not the prose. A well-written diagnosis that reaches the wrong cause
fails; a terse one that reaches the right cause passes.

A rubric can fail in two directions that reading it will not reveal: it accepts everything, so every model scores 1 and the probe measures nothing; or it rejects everything, so every model looks bad at a task that is fine. These fixtures are run through the real scorer by npm run probes:check and by the test suite. The correct answer must score 1, and every wrong answer must not.

The query returned no rows. `db.orders.findMany` returns `[]` when nothing matches, and
this account has no pending orders, so `orders[0]` is `undefined` — line 14 then reads
`.customer` on `undefined` and throws. Nothing is wrong with the data or the database.

Fix: handle the empty result before indexing.

    if (orders.length === 0) return renderEmptyInvoice(accountId);
    const customerName = orders[0].customer.name;

Optional chaining on `.customer` would stop the throw but leave `customerName` undefined,
moving the failure into `renderInvoice`.

The order record returned by the database is missing its `customer` relation, so
`orders[0].customer` is undefined and reading `.name` throws. This usually means the
customer join was not included in the query.

Fix: include the relation, or use optional chaining:

    const customerName = orders[0].customer?.name;

This is a transient database connectivity issue. `findMany` returned undefined because
the connection dropped mid-query, and the code then indexed into it. Add a retry with
backoff around the query and check the connection pool size.

`orders` is empty for this account, so `orders[0]` is undefined and line 14 throws.

Fix: `const customerName = orders[0]?.customer?.name;`

This could be caused by a few things: the orders array may be empty, the customer
relation may not be loaded, or the database may have returned an unexpected shape.
I would add logging around line 14 and defensive checks throughout to narrow it down.

v1 · current

Initial version.

maxTokens

1400

temperature

0

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-opus-5",
  "messages": [
    {
      "role": "system",
      "content": "You are a backend engineer debugging a production error. You reason from the evidence you are given and you do not assert causes the evidence does not support."
    },
    {
      "role": "user",
      "content": "Diagnose this production error in under 120 words. State the root cause and the fix.\n\nERROR\nTypeError: Cannot read properties of undefined (reading '\''customer'\'')\n    at buildInvoice (/srv/app/invoice.js:14:31)\n    at processBatch (/srv/app/batch.js:52:20)\n    at async main (/srv/app/index.js:9:3)\n\nCODE — /srv/app/invoice.js\n10  async function buildInvoice(accountId) {\n11    const orders = await db.orders.findMany({\n12      where: { accountId, status: '\''PENDING'\'' }\n13    });\n14    const customerName = orders[0].customer.name;\n15    return renderInvoice(customerName, orders);\n16  }\n\nWHAT WE KNOW\n- The account being processed when this threw has no pending orders.\n- db.orders.findMany returns an array, and returns [] when nothing matches.\n- The same code has run successfully for every other account in the batch.\n- No database errors were logged during the incident."
    }
  ],
  "max_tokens": 1400,
  "temperature": 0
}'