yeah, i get that. but i think what i’m more interested in is problem navigation.

once someone joins a company, they’re not always going to be working on things they’ve already seen before or can talk about from past experience. they’ll get dropped into unfamiliar systems, weird bugs, new constraints, and have to figure out what matters.

so part of the value to me is seeing how someone behaves when they don’t already know the terrain, where they look first, what assumptions they make, how they narrow things down, and how quickly they build enough understanding to make a good decision.

i think that’s probably part of why leetcode-style interviews survived this long too. for all their flaws, they give you a problem the candidate hasn’t necessarily seen (sometimes) in that exact form and let you watch how they navigate it.

and the non-technical execs are usually the worst for it 😭 to them AI is basically a magic button, so somehow you’re expected to ship 100x faster on something nobody has even properly defined yet.

but that’s more the chaos you learn to survive once you actually have the job.

i’m more focused on the bit before that, what people are actually going to be tested on to get through the interview in the first place, especially with the market being this rough.

isn't that only true if you treat the result as the entire interview?

i'm thinking of it more as a way to create evidence for the conversation. not “you passed 8/10 checks so you're good,” but show me how you approached something you hadn't seen before and let's talk through the decisions you made.

Why though? AI is already part of how a lot of engineers work, and people are already using it to game traditional coding interviews. so trying to enforce “pretend AI doesn’t exist” feels a bit like giving a kid a calculator every day, then deciding the exam should test whether they can hide from the calculator.

i’d rather accept that the tool exists and test the part that still matters: can they reason, verify, catch bad output, understand the codebase, and make a safe change?

the AI shouldn’t be the thing being tested. the engineer’s judgment should be.

you absolutely can, and honestly you probably should be able to.

my point is more that if the interview explicitly allows AI, then using it well becomes part of the skill being tested too. not “let AI do it for me,” but knowing when it helps, when it’s wrong, and when it’s faster to just do the thing yourself.

yeah, obviously 😭

i’m not saying AI replaces knowing how to code. i’m more interested in what comes after that baseline can you actually work through an unfamiliar repo, use AI without blindly trusting it, and make a safe change?

yeah, i get that concern.

i don't think the point should be “can this person produce code fastest with an llm.” that would be pretty bleak.

for me the interesting part is almost the opposite: can they still make good engineering decisions with AI in the loop? understand tradeoffs, reject bad suggestions, preserve the design, know when the generated fix is technically valid but wrong for the codebase.

if the assessment only rewards output, then yeah, it just turns people into cogs.

i think we're talking about slightly different things, i'm not suggesting the llm should decide whether someone is a good engineer or a good fit. i wouldn't trust that either.

the interesting part to me is putting someone in the environment they would actually work in, repo, ticket, tests, AI available and seeing how they reason through it.

do they understand what the AI gives them? question it? verify it? know where to look when it's wrong?

the conversation afterwards can still be the most important part. the repo just gives you something concrete to have that conversation about.

yeah, the platform exists.

this post is more me trying to validate the idea and hear what people would actually want from something like this before i over-engineer it.

i can share the link if you want to try it.

this comment actually sent me down a rabbit hole lol

the “score the trace, not just the diff” part is the bit i keep coming back to. the final fix is easy to measure, but the interesting signal is probably what they inspected, what they tried, whether they verified the AI suggestion, etc.

i’ve been building something around this since that thread and that’s still the part i haven’t figured out properly - Groundwork but i'm curious, how much of the investigation do you capture before it starts becoming fake/surveillance-y?