"You can sit anywhere on the spectrum from vibe coding to agentic engineering with the same agent. The thing that decides where you land is verification.The right spot on the spectrum depends on the stakes. The skill is knowing where to draw the line for each task.The right spot on the spectrum depends on the stakes. The skill is knowing where to draw the line for each task.There are two mechanisms. Tests cover the deterministic parts: this input, that output. Evals cover the parts that aren’t deterministic, and the paper splits them in a way I found useful. Output evaluation asks whether the final result is correct. Trajectory evaluation asks whether the path it took to get there, the tool calls and the reasoning, was sound. You want both. An answer that looks right but skipped its checks is more dangerous than one that’s obviously broken.If I had to hand a leader one line from the paper, it’s this: Set the bar at the eval, not the demo. A demo shows an agent can work once. An eval suite with a real rubric shows it works reliably. I keep making this argument; see “Agentic Code Review.”AI compresses the lifecycle, but unevenly, and the unevenness is the whole story. Implementation drops from weeks to hours. Requirements, architecture, and verification stay slow because they’re judgment work. So specification quality becomes the bottleneck, and verification moves to the middle."oreilly.com/radar/the-new-soft#AI #GenerativeAI #AIAgents #AgenticAI #AgenticEngineering # #VibeCoding #SDLC #SoftwareDevelopment #Programming

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 months ago

@remixtures the trajectory point is the one people skip. you can get the right output for the wrong reason, and if you only grade the outcome you'll never catch it, until the one time the shortcut doesn't line up. by then it's already in production.

  • source