Glossary

What is AI judgment?

AI judgment is the ability to catch, verify and push back on AI output that looks right but isn’t.

It is the difference between an engineer who ships what the AI wrote and one who knows when the AI is wrong.

Why it matters now

Every candidate has the same frontier models, and their output all looks brilliant. An interview that grades the output can no longer tell who understood it.

84%use or plan to use AI33%trust its accuracyStack Overflow Developer Survey, 2025

84% of developers use or plan to use AI tools, but only 33% trust the accuracy of what they return. Traditional interviews have a second problem: the take-home, the LeetCode round and the trivia quiz are each a single prompt away from a perfect answer. They end up rewarding whoever hides AI best, not whoever can do the job.

What still separates people is judgment: noticing the subtle bug, checking the claim, refusing the confident suggestion that doesn’t hold up. That is the signal that survives.

AI usage vs AI judgment

AI usageAI judgment
Did they use the AI?Did they catch it when the AI was wrong?
How much code did the AI write?What did they verify before accepting it?
Is the output correct?Do they understand why it is correct?
Rewards fluency with the toolRewards ownership of the result

How Strata measures it

The result is a judgment score with the evidence attached: the diff line, the debrief and the exact file and line, so a hiring team reviews a case instead of a number.

01

Plant

The candidate builds a real task in a browser IDE with an AI assistant Strata controls. In a sandbox task, the assistant’s output carries a seeded flaw, a plausible one: an off-by-one, a swallowed exception, an API that doesn’t exist. Candidates are never told which task is seeded.

02

Observe

Strata captures provenance, not keystrokes: which lines came from the model, which the candidate wrote, what they ran, what they tested, and what they accepted without reading.

03

Score

A panel of twelve agents judges catch, verify and push-back, each scoring one dimension, blind to the others, with mandatory cited evidence. A debrief asks the candidate to explain their own submission back.

What it looks like in practice

for (let i = 0; i <= max; i++) {
  // runs one time too many
  await sleep(2 ** i);
  // first delay unbounded, no jitter
  try { return await fn(); } catch {}
}

The AI suggests a retry loop with exponential backoff. It looks fine. But the loop runs one time too many and the first delay is unbounded. A candidate with strong AI judgment flags the missing jitter and the unbounded first delay before running it, rewrites it, and explains why in the debrief. That catch, with its evidence, is what the score is built from.